Scalability Plan

Okay. Here's the plan.

A little while ago, I put in the notion of the requested subcontext, which holds the data actually requested to be there. From this, the overlay (noinherit) and normal subcontexts are built (hoorah.)

So my aim is to drop the latter two, so that those are built on the fly, on demand. We maintain a list somewhere of configurable entries held in memory, to cut load-in costs.

Then, we shift the values into bdb - these are, by far, the chunkiest bits of memory.

Then, the datasets themselves get shifted, probably as ordered clouds of attributes.

Finally, we handle the datastore itself by heirarchical searches through the datasets - actually this could be done first - and attempt to drop the paths inside the datasets.

Overall, this should mean that the data is held on BDB, and actually transactionally controlled by it, but also we still use the advantages of the in-memory processing - namely speed.

For smallish systems, it shouldn't impact at all.


Dave Cridland
Last modified: Mon Nov 29 23:14:41 GMT 2004