Next week, I’ll see if the incrementalization actually helps us scale.

Only had a tiny bit of time today; I worked more on the least common ancestor query. Here is what new work contributed to better scaling:

• Incrementalizing the code (duh) did in fact let me scale it farther. The prior program was failing because of memory problems in a large final join; breaking down that join into multiple steps, one at a time, inside the loop reduces the memory crunch.

• Obvious Raco optimization: turn X = X + Y; into X += Y;, which is a big deal when X and Y both come from and are inserted into a database. (Here, + is UnionAll).

  The only reason this optimization had not been implemented before is that this was never actually the bottleneck.

Once again, it goes to show what working with real data and real queries can do for your system’s applicability.