Brian Warner dropped by today and we spent a good chunk of the day speccing
out New PB. The New PB will resolve pretty much all the security issues I've
come up with in the long purgatory of implementation-less contemplation of
my previous work, plus it gives us a good place to define all *kinds* of
wacky transport semantics that would otherwise have made the old
infrastructure blow up.
In brief, every object that you want to publish remotely will have a schema
or interface that specifies how it is serialized or referenced remotely. We
will be trying to produce arbitrarily publish-able objects that have both
state and methods, where you can say how you want to send things by default,
but you can also switch it up in different situations without defining a
bazillion classes.
The only lack of convenience is that we are going to spit out a
near-continuous stream of warnings if you try to go for the "arbitrary
graphs of objects" behavior that the previous version allowed. Sorry, folks,
but that's just not safe in python and it can never be made safe :).
However, we will still allow it for prototyping purposes, and maybe even add
a mode where the system spits out an example schema after you run "trial"
over unit tests that invoke the PB serializer on a particular set of
classes.
We also discussed his "Pet Mail" project, which is a replacement for SMTP.
It got me thinking about how poorly defined our shared vision for Q2Q is.
We've really only started the very basic design discussions. Brian's project
is a really good example of the kind of straightforward crypto work that we
haven't had any time or inclination to do. However, it's (explicitly) got no
deployment considerations, so if we can leverage some of the planned
features of Q2Q and Quotient to get the base PetMail protocol widely
deployed, we can establish a shared, clear basis for verified personal
communication. Divmod can build more interesting applications on top of
that, but it would give us a simplified transport layer to sell to other
ISPs so that we don't need to spend all our processing power and disk on
brute-force spam filtering.
Wow, it's been a long day.
This has definitely been my best coding day on Divmod yet this month. Into the wee hours of the morning, Allen and I tore through the rest of the tasks which I had estimated it would take the rest of the release to complete.
We also had a fun discussion about the successor to twisted.world, CARCOSA (backronymed to "Concurrent, Atomic, Reliable Object Storage Architecture"). When we can refactor Quotient's store into a bit more general of a structure, and also take advantage of what must be a significant speed boost in using fixed-length rather than variable records in bsddb, we should be able to implement everything that we had hoped for in twisted.world and possibly more.
We're probably going to need to add a twisted.python.schema module, which should be shared between Formless, CARCOSA and PB. While CARCOSA is dealing about storage and Formless/PB is dealing with interactivity, you should be able to express type constraints the same way.
An object which implements a TypedInterface and also specifies __schema__ itself will be able to be published on the web, transactionally stored in a database, and also published over a custom, interactive protocol, all with no additional work besides what the average Java programmer has to endure when defining a class. Plus, you can develop your objects unencumbered by the schema and only nail them down once you've developed an understanding of their requirements through experimentation.
This has definitely been my best coding day on Divmod yet this month. Into the wee hours of the morning, Allen and I tore through the rest of the tasks which I had estimated it would take the rest of the release to complete.
We also had a fun discussion about the successor to twisted.world, CARCOSA (backronymed to "Concurrent, Atomic, Reliable Object Storage Architecture"). When we can refactor Quotient's store into a bit more general of a structure, and also take advantage of what must be a significant speed boost in using fixed-length rather than variable records in bsddb, we should be able to implement everything that we had hoped for in twisted.world and possibly more.
We're probably going to need to add a twisted.python.schema module, which should be shared between Formless, CARCOSA and PB. While CARCOSA is dealing about storage and Formless/PB is dealing with interactivity, you should be able to express type constraints the same way.
An object which implements a TypedInterface and also specifies __schema__ itself will be able to be published on the web, transactionally stored in a database, and also published over a custom, interactive protocol, all with no additional work besides what the average Java programmer has to endure when defining a class. Plus, you can develop your objects unencumbered by the schema and only nail them down once you've developed an understanding of their requirements through experimentation.
Well, it took most of the day, but once Allen pointed out a stupid error I
was making by ignoring a potential error condition, the first pass on
bsddb/store integration is already committed to Quotient! Plus, I took a nap
today, so I should have a few more good hours of programming left to get
itempool plugged in.
In keeping with the theme of this journal, I decided to update the picture.
Some of you may recognize the icon associated with this post :).
Strangely it did not work until I uploaded an LZW-compressed GIF. It didn't work as either a PNG output from Gimp, from pngcrush, from convert or from sodipodi, nor a JPG from convert or gimp. I suppose you have to mix the evil of the Sign itself with the evil of the Unisys patent
Strangely it did not work until I uploaded an LZW-compressed GIF. It didn't work as either a PNG output from Gimp, from pngcrush, from convert or from sodipodi, nor a JPG from convert or gimp. I suppose you have to mix the evil of the Sign itself with the evil of the Unisys patent
Well! It turns out that the bsddb conversion was pretty easy after all. (At
least, moving object storage to it.) Now all I have to do is track down this
stupid identity management issue and I'm all set.
The only problem is that somehow, objects are sometimes being doubly instantiated when they should only be instantiated once, and sometimes they're not being re-instantiated when they should be garbage collected and re-created.
In the particular issue I'm debugging at the moment, the object relationship is SUPPOSED to be:
stack : item
stack : pool : store : weak cache : item
and since the stack is holding a strong reference to the item, it shouldn't go away and be re-created, but somewhere in there there's a mistake.
Is there any common pattern, either for implementing or debugging these kinds of "this object can be garbage collected back to storage but it really only exists once" kind of things?
The only problem is that somehow, objects are sometimes being doubly instantiated when they should only be instantiated once, and sometimes they're not being re-instantiated when they should be garbage collected and re-created.
In the particular issue I'm debugging at the moment, the object relationship is SUPPOSED to be:
stack : item
stack : pool : store : weak cache : item
and since the stack is holding a strong reference to the item, it shouldn't go away and be re-created, but somewhere in there there's a mistake.
Is there any common pattern, either for implementing or debugging these kinds of "this object can be garbage collected back to storage but it really only exists once" kind of things?