Well! It turns out that the bsddb conversion was pretty easy after all. (At
least, moving object storage to it.) Now all I have to do is track down this
stupid identity management issue and I'm all set.
The only problem is that somehow, objects are sometimes being doubly
instantiated when they should only be instantiated once, and sometimes
they're not being re-instantiated when they should be garbage collected and
re-created.
In the particular issue I'm debugging at the moment, the object relationship
is SUPPOSED to be:
stack : item
stack : pool : store : weak cache : item
and since the stack is holding a strong reference to the item, it shouldn't
go away and be re-created, but somewhere in there there's a mistake.
Is there any common pattern, either for implementing or debugging these
kinds of "this object can be garbage collected back to storage but it really
only exists once" kind of things?
Today I actually discovered BSDDB. I can't believe I missed this.
It's an efficient, in-process, open-source, simple, transactional database
which does almost everything I have ever wanted a database to do. For
free.
I read the transaction store documentation at sleepycat's site, and discovered that bsddb does things I hadn't even considered - for example, it is possible to transactionally move a non-database file using the bsddb API. I realize that this may not mean much to anyone but me, but I was so moved by that possibility that I nearly cried.
This discovery is both exciting and terrifying. Allen and I ran into this together, pair programming and attempting to reconcile the highly single-process logic in the current Quotient with the highly multi-process logic in the new QQ (Quotient Queues) module we're integrating. After looking at one particular function that was obviously fragile and could lose data at several points, we decided we needed to solidify and centralize our transaction processing into a single place. Since we knew that BSDDB had "some transaction support", we figured we could use it for what we needed.
We got more than we bargained for. One of the first things that we discovered that we had previously mis-read the documentation: we believed that BSDDB was single-process, based on the fact that their documentation talked about multi-reader access only being available for non-transactional data stores, and other areas referred to "threads of control".
It turns out that it's perfectly usable from multiple processes simultaneously. "Thread of control" actually means "process, thread, or other encapsulation of a program counter" the way they use it in their documentation.
So, on the one hand, it will make the incredibly arduous task of making our central data store 100% reliable much, much easier than it previously was. Rather than being an ongoing task with many threads left hanging, it will be almost completely done when we finish this refactoring. On the other, this increases the amount of changes we have to get done for this release - by saturday. This is a much larger snag than I expected to hit at this point, considering that we'd already gotten through a lot of the "hard stuff" - figuring out how to make multi-process communication both transactional and observable from the user interface. (Luckily, much of that work won't be wasted since we will be using it to manage transactions across multiple machines.)
I read the transaction store documentation at sleepycat's site, and discovered that bsddb does things I hadn't even considered - for example, it is possible to transactionally move a non-database file using the bsddb API. I realize that this may not mean much to anyone but me, but I was so moved by that possibility that I nearly cried.
This discovery is both exciting and terrifying. Allen and I ran into this together, pair programming and attempting to reconcile the highly single-process logic in the current Quotient with the highly multi-process logic in the new QQ (Quotient Queues) module we're integrating. After looking at one particular function that was obviously fragile and could lose data at several points, we decided we needed to solidify and centralize our transaction processing into a single place. Since we knew that BSDDB had "some transaction support", we figured we could use it for what we needed.
We got more than we bargained for. One of the first things that we discovered that we had previously mis-read the documentation: we believed that BSDDB was single-process, based on the fact that their documentation talked about multi-reader access only being available for non-transactional data stores, and other areas referred to "threads of control".
It turns out that it's perfectly usable from multiple processes simultaneously. "Thread of control" actually means "process, thread, or other encapsulation of a program counter" the way they use it in their documentation.
So, on the one hand, it will make the incredibly arduous task of making our central data store 100% reliable much, much easier than it previously was. Rather than being an ongoing task with many threads left hanging, it will be almost completely done when we finish this refactoring. On the other, this increases the amount of changes we have to get done for this release - by saturday. This is a much larger snag than I expected to hit at this point, considering that we'd already gotten through a lot of the "hard stuff" - figuring out how to make multi-process communication both transactional and observable from the user interface. (Luckily, much of that work won't be wasted since we will be using it to manage transactions across multiple machines.)
Just finished surveying my work for the day and writing my update about it.
It's been a long day and it looks to be one again tomorrow.
Musing about blogging, though. It never really occurred to me how the format blends somewhere between a mailing list and a chat room. The logjam client really brings that out, since the icon for updating the blog is right there on my panel all the time.
I suppose I should decide something to do with this blog. Considering it is likely to be a temporary experiment, I don't want to try for anything terribly ambitious. I will keep the parameters loose, but I think I'll go for a combination of personal rambilng and a discussion of the design I'm doing at work.
On that note, today I did a lot of thinking about context, which has something to do with blogs. One of the reasons - the only real reason, I'd posit - that blogs are so popular is that email software is so terrible. I should easily be able to put a "new email message to list" button onto my gnome panel, but this never occurred to me, and it's more work than it's worth. (I did work out that a launcher icon that runs "mozilla -remote 'openURL(mailto:twisted-python@twistedmatrix.com)'" will do the trick, but really, if I can't work this out on my own, is the average user likely to? Is this even possible on non-UNIX OS'es? with non-mozilla mail clients? COM doesn't count unless you can do it with the tools generally available with free email clients...)
In principle, though, there is nothing more sophisticated going on here than an email with a couple of X-headers and a cute trick with xmms - not even my preferred mp3 player any more, though I switched back to it just to see the 'detect' button suck the song titles out of it.
But, back to context. The thing about email software that's so terrible is that it presents all messages to you at equal priority. It's a considerable amount of work to re-prioritize messages in a meaningful way, and even if you do, it's very difficult to track your time when working with email to spend the appropriate amount of time dealing with them.
Blogs solve the first problem very well. They don't stack up on your inbox. When you go to read a blog, you are quite explicitly in your "reading a blog" context. Even without any time tracking tools, you know approximately how much time you ought to spending doing that, so they do decently well with the second. The poor UI characteristics of the web don't evidence themselves so much because you're just statically reading, and the web does have very good layout tools.
Web forums don't fare so well. They do have a noticeable disadvantage against email clients because you can't consolidate your messages, quoted reply is difficult and broken and inconsistent among implementations. They are much worse at solving the second problem, too - it's very easy to get sucked into a forum discussion which takes hours to compose a scathing reply when you should be dealing with other forms of input into your life.
Personally, my vision for Quotient involves a consistent, context-based way of reading messages and holding online conversations with integrated timetracking and task management. When I sit down to read a blog, I will type "I am giving myself an hour to read this today", and at the end of that hour, I want my browser window to be closed and any messages I'm working on to be automatically saved (with undo buffer and clipboard, if possible, because I might have just cut something big out of the message I was editing). A subtle reminder to get on with my life and not get lost online. However, there is no distinction for me between a blog and a mailing list; I want the same time-limiting and monitoring to take place there, including my reply compositions. I can always click the "give me an unbounded amount of time to noodle around on the web" button, but then I should be able to see a visible indication of how much time I've spent.
Tools like this would serve a greater purpose than simply inhibiting 'net addiction - they would give me a way to continue to enjoy and participate in the communities I frequent in a more substantial way without interfering with real life.
In that spirit, I will leave this message unedited so that I don't spend any more of the time I should use for sleeping writing it :).
Musing about blogging, though. It never really occurred to me how the format blends somewhere between a mailing list and a chat room. The logjam client really brings that out, since the icon for updating the blog is right there on my panel all the time.
I suppose I should decide something to do with this blog. Considering it is likely to be a temporary experiment, I don't want to try for anything terribly ambitious. I will keep the parameters loose, but I think I'll go for a combination of personal rambilng and a discussion of the design I'm doing at work.
On that note, today I did a lot of thinking about context, which has something to do with blogs. One of the reasons - the only real reason, I'd posit - that blogs are so popular is that email software is so terrible. I should easily be able to put a "new email message to list" button onto my gnome panel, but this never occurred to me, and it's more work than it's worth. (I did work out that a launcher icon that runs "mozilla -remote 'openURL(mailto:twisted-python@twistedmatrix.com)'" will do the trick, but really, if I can't work this out on my own, is the average user likely to? Is this even possible on non-UNIX OS'es? with non-mozilla mail clients? COM doesn't count unless you can do it with the tools generally available with free email clients...)
In principle, though, there is nothing more sophisticated going on here than an email with a couple of X-headers and a cute trick with xmms - not even my preferred mp3 player any more, though I switched back to it just to see the 'detect' button suck the song titles out of it.
But, back to context. The thing about email software that's so terrible is that it presents all messages to you at equal priority. It's a considerable amount of work to re-prioritize messages in a meaningful way, and even if you do, it's very difficult to track your time when working with email to spend the appropriate amount of time dealing with them.
Blogs solve the first problem very well. They don't stack up on your inbox. When you go to read a blog, you are quite explicitly in your "reading a blog" context. Even without any time tracking tools, you know approximately how much time you ought to spending doing that, so they do decently well with the second. The poor UI characteristics of the web don't evidence themselves so much because you're just statically reading, and the web does have very good layout tools.
Web forums don't fare so well. They do have a noticeable disadvantage against email clients because you can't consolidate your messages, quoted reply is difficult and broken and inconsistent among implementations. They are much worse at solving the second problem, too - it's very easy to get sucked into a forum discussion which takes hours to compose a scathing reply when you should be dealing with other forms of input into your life.
Personally, my vision for Quotient involves a consistent, context-based way of reading messages and holding online conversations with integrated timetracking and task management. When I sit down to read a blog, I will type "I am giving myself an hour to read this today", and at the end of that hour, I want my browser window to be closed and any messages I'm working on to be automatically saved (with undo buffer and clipboard, if possible, because I might have just cut something big out of the message I was editing). A subtle reminder to get on with my life and not get lost online. However, there is no distinction for me between a blog and a mailing list; I want the same time-limiting and monitoring to take place there, including my reply compositions. I can always click the "give me an unbounded amount of time to noodle around on the web" button, but then I should be able to see a visible indication of how much time I've spent.
Tools like this would serve a greater purpose than simply inhibiting 'net addiction - they would give me a way to continue to enjoy and participate in the communities I frequent in a more substantial way without interfering with real life.
In that spirit, I will leave this message unedited so that I don't spend any more of the time I should use for sleeping writing it :).
Well! Thanks to amir I now have an invited livejournal account, so I figured
I'd try it out and post something.