Today I actually discovered BSDDB. I can't believe I missed this.
It's an efficient, in-process, open-source, simple, transactional database
which does almost everything I have ever wanted a database to do. For
free.
I read the transaction store
documentation at sleepycat's site, and discovered that bsddb does things
I hadn't even considered - for example, it is possible to
transactionally move a non-database file using the bsddb API. I realize
that this may not mean much to anyone but me, but I was so moved by that
possibility that I nearly cried.
This discovery is both exciting and terrifying. Allen and I ran into this
together, pair programming and attempting to reconcile the highly
single-process logic in the current Quotient with the highly multi-process
logic in the new QQ (Quotient Queues) module we're integrating. After
looking at one particular function that was obviously fragile and could lose
data at several points, we decided we needed to solidify and centralize our
transaction processing into a single place. Since we knew that BSDDB had
"some transaction support", we figured we could use it for what we
needed.
We got more than we bargained for. One of the first things that we
discovered that we had previously mis-read the documentation: we believed
that BSDDB was single-process, based on the fact that their documentation
talked about multi-reader access only being available for non-transactional
data stores, and other areas referred to "threads of control".
It turns out that it's perfectly usable from multiple processes
simultaneously. "Thread of control" actually means "process, thread, or
other encapsulation of a program counter" the way they use it in their
documentation.
So, on the one hand, it will make the incredibly arduous task of making our
central data store 100% reliable much, much easier than it previously was.
Rather than being an ongoing task with many threads left hanging, it will be
almost completely done when we finish this refactoring. On the other, this
increases the amount of changes we have to get done for this release - by
saturday. This is a much larger snag than I expected to hit at this point,
considering that we'd already gotten through a lot of the "hard stuff" -
figuring out how to make multi-process communication both transactional and
observable from the user interface. (Luckily, much of that work won't be
wasted since we will be using it to manage transactions across multiple
machines.)