This week I've been in New York City with the rest of the divmod developers, trying to get our product
(service, really) into an acceptable state to start selling it to
people.
Aside from some
really unfortunate personal issues, this has been easily the best week
of the last year. Every day I sat down to do some work, every day I got
something quantifiable done. There are design issues, but there was no
thrashing between completely different ways of doing things, only subtle
corrections of possible problems.
Even the one really frightening thing that happened this week, a DB_RUNRECOVERY
error which spuriously appeared when we attempted to open our database file,
turned out to be a peculiarity in the interface of bsddb and not a systematic problem. JP and I managed to get the
database re-opened and uncorrupt just by reading the mnet source code and
tweaking some variables.
The two stars of this week were really "atop" (atomic transactional object
persistence), and "nevow" (pronounced "nouveau", the next version of woven).
After a week of intense work, we have a new UI and a new database: the new
UI is almost to the point where the old one was, and the new database is far
beyond the point where the old one was.
Despite this, there's a huge amount of work remaining, especially to rewrite
our input handling and such. Still, I estimate at this point that I'll be
keeping my email in this system in 2 weeks or less, and I'm far more sure of
that estimate than I have been at any point in the past.
Brian Warner dropped by today and we spent a good chunk of the day speccing
out New PB. The New PB will resolve pretty much all the security issues I've
come up with in the long purgatory of implementation-less contemplation of
my previous work, plus it gives us a good place to define all *kinds* of
wacky transport semantics that would otherwise have made the old
infrastructure blow up.
In brief, every object that you want to publish remotely will have a schema or interface that specifies how it is serialized or referenced remotely. We will be trying to produce arbitrarily publish-able objects that have both state and methods, where you can say how you want to send things by default, but you can also switch it up in different situations without defining a bazillion classes.
The only lack of convenience is that we are going to spit out a near-continuous stream of warnings if you try to go for the "arbitrary graphs of objects" behavior that the previous version allowed. Sorry, folks, but that's just not safe in python and it can never be made safe :). However, we will still allow it for prototyping purposes, and maybe even add a mode where the system spits out an example schema after you run "trial" over unit tests that invoke the PB serializer on a particular set of classes.
We also discussed his "Pet Mail" project, which is a replacement for SMTP. It got me thinking about how poorly defined our shared vision for Q2Q is. We've really only started the very basic design discussions. Brian's project is a really good example of the kind of straightforward crypto work that we haven't had any time or inclination to do. However, it's (explicitly) got no deployment considerations, so if we can leverage some of the planned features of Q2Q and Quotient to get the base PetMail protocol widely deployed, we can establish a shared, clear basis for verified personal communication. Divmod can build more interesting applications on top of that, but it would give us a simplified transport layer to sell to other ISPs so that we don't need to spend all our processing power and disk on brute-force spam filtering.
In brief, every object that you want to publish remotely will have a schema or interface that specifies how it is serialized or referenced remotely. We will be trying to produce arbitrarily publish-able objects that have both state and methods, where you can say how you want to send things by default, but you can also switch it up in different situations without defining a bazillion classes.
The only lack of convenience is that we are going to spit out a near-continuous stream of warnings if you try to go for the "arbitrary graphs of objects" behavior that the previous version allowed. Sorry, folks, but that's just not safe in python and it can never be made safe :). However, we will still allow it for prototyping purposes, and maybe even add a mode where the system spits out an example schema after you run "trial" over unit tests that invoke the PB serializer on a particular set of classes.
We also discussed his "Pet Mail" project, which is a replacement for SMTP. It got me thinking about how poorly defined our shared vision for Q2Q is. We've really only started the very basic design discussions. Brian's project is a really good example of the kind of straightforward crypto work that we haven't had any time or inclination to do. However, it's (explicitly) got no deployment considerations, so if we can leverage some of the planned features of Q2Q and Quotient to get the base PetMail protocol widely deployed, we can establish a shared, clear basis for verified personal communication. Divmod can build more interesting applications on top of that, but it would give us a simplified transport layer to sell to other ISPs so that we don't need to spend all our processing power and disk on brute-force spam filtering.
Wow, it's been a long day.
This has definitely been my best coding day on Divmod yet this month. Into the wee hours of the morning, Allen and I tore through the rest of the tasks which I had estimated it would take the rest of the release to complete.
We also had a fun discussion about the successor to twisted.world, CARCOSA (backronymed to "Concurrent, Atomic, Reliable Object Storage Architecture"). When we can refactor Quotient's store into a bit more general of a structure, and also take advantage of what must be a significant speed boost in using fixed-length rather than variable records in bsddb, we should be able to implement everything that we had hoped for in twisted.world and possibly more.
We're probably going to need to add a twisted.python.schema module, which should be shared between Formless, CARCOSA and PB. While CARCOSA is dealing about storage and Formless/PB is dealing with interactivity, you should be able to express type constraints the same way.
An object which implements a TypedInterface and also specifies __schema__ itself will be able to be published on the web, transactionally stored in a database, and also published over a custom, interactive protocol, all with no additional work besides what the average Java programmer has to endure when defining a class. Plus, you can develop your objects unencumbered by the schema and only nail them down once you've developed an understanding of their requirements through experimentation.
This has definitely been my best coding day on Divmod yet this month. Into the wee hours of the morning, Allen and I tore through the rest of the tasks which I had estimated it would take the rest of the release to complete.
We also had a fun discussion about the successor to twisted.world, CARCOSA (backronymed to "Concurrent, Atomic, Reliable Object Storage Architecture"). When we can refactor Quotient's store into a bit more general of a structure, and also take advantage of what must be a significant speed boost in using fixed-length rather than variable records in bsddb, we should be able to implement everything that we had hoped for in twisted.world and possibly more.
We're probably going to need to add a twisted.python.schema module, which should be shared between Formless, CARCOSA and PB. While CARCOSA is dealing about storage and Formless/PB is dealing with interactivity, you should be able to express type constraints the same way.
An object which implements a TypedInterface and also specifies __schema__ itself will be able to be published on the web, transactionally stored in a database, and also published over a custom, interactive protocol, all with no additional work besides what the average Java programmer has to endure when defining a class. Plus, you can develop your objects unencumbered by the schema and only nail them down once you've developed an understanding of their requirements through experimentation.
Well, it took most of the day, but once Allen pointed out a stupid error I
was making by ignoring a potential error condition, the first pass on
bsddb/store integration is already committed to Quotient! Plus, I took a nap
today, so I should have a few more good hours of programming left to get
itempool plugged in.
In keeping with the theme of this journal, I decided to update the picture.
Some of you may recognize the icon associated with this post :).
Strangely it did not work until I uploaded an LZW-compressed GIF. It didn't work as either a PNG output from Gimp, from pngcrush, from convert or from sodipodi, nor a JPG from convert or gimp. I suppose you have to mix the evil of the Sign itself with the evil of the Unisys patent
Strangely it did not work until I uploaded an LZW-compressed GIF. It didn't work as either a PNG output from Gimp, from pngcrush, from convert or from sodipodi, nor a JPG from convert or gimp. I suppose you have to mix the evil of the Sign itself with the evil of the Unisys patent