Progress is now visible in a more public location:
the unified Divmod SVN repository at http://divmod.org/svn/Divmod/trunk/I would have said "rewrite" rather than "refactor", but of course Everybody knows that's stupid. Plus, we are mostly migrating our old code base and cleaning it up along the way; the only component getting a complete rewrite is Atop - the rewrite of which is so fundamentally better that we came up with a better name: "Axiom".
To clarify the naming situation: the new Divmod repository has Axiom and a new version of Mantissa, Python package name 'xmantissa' to avoid package name conflicts during the transition period. eventually it will contain a package for Quotient, Sigma, and several other things. During the transition the Python package names will all start with 'x' but I will still refer to them by their project names, since the older projects will go away and the module names will eventually change.
Initially I was very concerned when we began the experiment that lead to this code revolution. It began, as bad ideas are wont to do, as a joke. I mentioned some client work that we are doing to JP, (who had already been rewriting some things) and my difficulty in choosing an appropriate persistence solution given some of the maintenance issues we'd been having with Atop.
Exasperated after an hour of discussion, JP said, "Why don't you just use SQLite". Now, I'd looked into it some time ago and (the ostensible punchline of the joke) it was garbage. However, that was SQLite version 1, not SQLite3, which has a different API and several critical features that made it considerably more appropriate to our tasks.
A few hours later I had a working prototype of maybe half of the functionality from our existing database. I was suitably impressed; SQLite was giving us all the benefits of SQL (ad-hoc queries and indexing, relational operations on data, a fast query engine) without any of the drawbacks (difficult to customize, unportable server, fragile and time-consuming deployment). I realized that we had got something radical on multiple levels.
In the past I've been very conservative about telling people when and whether to use Divmod's open-source released software, With this new system, I say: jump in. Use it. Only a week into implementation it might be a bit premature to launch a production system with it just yet, but indications are very positive that we will be able to do just that within the month.
The code is shorter, clearer to read, easier to maintain, and the database is Pickle Free℠.
(Okay, it's not really a service mark, but it should be. Pickle is the winner for causing problems for us.)
We are building from our experience with 5 previous persistence systems, 3 previous plug-in frameworks, and 4 previous authentication databases. A curious side-effect of all that experience - and the effect that the title is referring to - is that certain development methodology concepts become irrelevant. Most notably, "YAGNI", from XP, is no longer of any use: we know exactly how much extensibility we need. At every point in implementing this system we have known whether to fuse a component together because we'd built unnecessary additional complexity into previous systems, and where to use a plug-in architecture because we'd needed to inject ugly code into the middle of a monolithic routine.
As a result, where our architecture was heavily monolithic before, now it is almost entirely composed of plugins. It is so plugin-happy, in fact, that there is a database with Service plugins in it, which activate when the database is started from twistd; it contains its own configuration, including port-numbers, so nothing need live in a text configuration file. The web application system is built around this as well; so there is a plugin lookup for invoking raw nevow IResource implementors without sessions (for example, for XML APIs), then IResource implementors which do require a session, then IResource implementors which are specific to a particular user, and finally Fragment instances which plug into a generic hierarchical navigation system. At each level there is a distinct and clear place to put new plugin code, and large portions of it are self-similar. For example, you can install the hierarchical navigation both onto a public site and onto a user's private application pages, since the "web server" implementation is an IResource which can be installed either onto a toplevel database, or any user's personal database.
Oh, and did I mention - LivePage support built right in?
The net result of this is that you can build themeable, multi-user web applications with the code that's in SVN right now. The example isn't visually appealing, but the code is nice, and it's composed from a stack of plugins.
I'm very excited about the possibilities of what we'll be transforming our system and our application into within the next few weeks. I'd like to invite everyone who has been interested in Divmod's open source work in the past to have a look at the new repository, and consider coming to #divmod on irc.freenode.net to look for something to hack on. Considering the higher-level and easier-to-understand nature of the implementation of Axiom vs. Atop, I would also love it if we could find some people to help us document it right from the start.
So - anybody out there looking for an open source project's website to maintain?
