... and for some reason, I'm awake again.

One last thing for tonight.

I have made a few enhancements to HATE since I announced it. I thought that since my previous post featured a screenshot of it, I should make those changes available. It is actually usable now, since I bound C-S-n to 'new window' as it is in gnome-terminal.

You can download the new version here.

By the way - sorry for the RSSSpam, but I am trying to convey a certain feeling here, to invoke a sense of the pace of development at Divmod and my general enthusiasm about the new code we, and I, have been producing over the last few weeks.

Is an adjective forming in your head? Good. No? Okay.

The adjective is furious.

Update: Relax, folks. The implied noun phrase here is pace of development, see above. It's not "at you".

Did you catch my reference to David Heinemeier Hansson in the previous article? It wasn't simply the unmasked, raw, hideous envy of his brilliant success - it was foreshadowing. That's because I am like a word ninja, assaulting you with various literary conventions.

You better watch out. I could, quite possibly, be on a rampage, what with all these releases, and writing about them.

I have been studying the new crop of web frameworks to see what the fuss is all about - watching screencasts, reading weblogs. At first I found the whole thing sort of boring. I could not understand what people got so excited about. I was over-focusing on the (rather boring) problem domain that all these frameworks attack - blogs, wikis, to-do lists and so on, and just not understanding why it was all so compelling.

This week I took a step back and realized it was not the product, but the process that made it so interesting. It's not that you're making a wiki - it's that it is so incredibly easy to make a wiki with TurboGears that, if you are already the sort of person who already finds wikis interesting and wants to explore the far-reaching problem domains associated with wikis, the ease with which you can sail past the easy stuff that everyone already agrees upon must be absolutely incredible. There are some really great lessons for framework developers that can be learned just by watching how these things are taught, rather than paying attention to what they actually do.

This release of Mantissa took the first small, but determined steps to make using Twisted, and the whole Divmod suite, more immediately approachable. There isn't any stub or automatic code generation happening yet, but we did improve the plug-in for the 'axiomatic' command-line tool quite a bit. Here is a screenshot - a pre-screencast, if you will:

That simple set of commands will create a database with a webserver, login system, an account for "admin@localhost" with a password of your choosing, a through-the-web interactive AJAX Python command line, and enough scaffolding at least on the database side for you to start deploying Mantissa plug-ins. Here's a peek:


Things will only improve from here on out.

I don't have any illusions that Mantissa will supplant TurboGears or Django or Rails or whatever as the tool du jour. We're not focused on wikis or blogs. We'll probably implement one, for integration between those features and the other stuff we're implementing, but it might still be harder to write a blog using Axiom and Mantissa than using one of the aforementioned tools.

Still, I guarantee it will be easier to make an Internet telephone service using Mantissa. Or, for that matter, a chat server that does IRC and Jabber. Or a clan server that controls instances of hl2 and quake4 multiplayer daemons.

I do certainly hope that we will get at least some press for our efforts, but to be honest the prospect of relative obscurity doesn't bother me. It might even be a sign of a good thing. It's similar to the bike shed problem: everybody knows how to build a bike shed, or a blog, so there's a huge demand for really good, really cheap tools to build them.

Still, dear reader, if you will permit me a moment of hubris as I labor down in the spice mines: somebody has to provide the power that those tools run on, so ultimately there's a demand for that nuclear power plant, too.

You see how I took that last bit into a comparison, without using 'like' or 'as'? Blam, it's a metaphor. Caught you by surprise there, didn't I! My writing technique is unstoppable.

We released Axiom again today.

It's making progress at an even keel these days.

A few minor issues: some bugs were fixed in upgraders, some deployment order dependency issues with Epsilon were fixed. We added some more data integrity checks, and came up with a way to dodge a few more inheritance quirks with Item.

The major upgrade in this release is that queries are objects, rather than simply generators. This shift in API is a subtle step further towards exposing as many efficient SQL operations as possible without exposing any SQL.

Since the release announcement is already over, you might have guessed - I tricked you. This is hardly a release announcement at all. It's an explanation of one of Axiom's goals: complete encapsulation of SQL in an object-oriented model.

Some people, most notably David Heinemeier Hansson, would say that's a pretty bad goal to have. Oddly enough, with the people they are generally arguing against, I'd agree. The relational model should get some respect. It's where your data actually lives. You shouldn't swaddle it to the point where it's suffocating.

The agreement ends pretty quickly though. I think of SQL in your application like fire on a cold night. It keeps me warm, sure, and I shouldn't douse it with water because I'm afraid of getting burned. I might freeze. That doesn't mean I stick my hand into the open flame.

You might ask, why such an intense metaphor? Isn't SQL just another tool in my programming toolbelt?

Metaprogramming is hard, and dress it up however you like, that's what using SQL is. Your code is generating other code, and evaluating its results.

Because metaprogramming is so hard, it is almost exclusively the province of frameworks, environments and operating systems. For good reason, too. Whenever code generates other code, there are potentially very serious mistakes that can get made. SQL is a hobbled language in most databases, so the damage is restricted. It's only restricted in the sense that it won't cause your server to segfault, though. Your database is probably where 99% of your application's (and possibly your company's) value lives anyway, not your server's heap memory. Mistakes in generating SQL that talks to that database can be very costly.

Getting away from SQL injection attacks is a minor feature of Axiom, so it's easy to forget that it is a really serious problem. People make this mistake all the time, with disastrous consequences. Unless you read Bugtraq, that is. In which case you are probably reminded of this somewhere between 2 and 20 times per week.

Having an object model for your SQL also helps you generalize things that might otherwise be overspecific, and test unrelated database features more independently. For example: there is a utility function in Axiom (go read the docstring, it's fun) which finds overlapping ranges of values. The query looks like this:

    OR(
        AND(startAttribute >= startValue,
            startAttribute <= endValue),
        AND(endAttribute >= startValue,
            endAttribute <= endValue),
        AND(startAttribute <= startValue,
            endAttribute >= endValue)
        )

If that had been accomplished with hand-generated SQL rather than objects, I'd have to quote every one of those attributes. I'd have to figure out how to make sure that the table's name was fully qualified whenever the user passed data into this function - if it needed to be. It would be hard to write a test for. Finally, it'd be hard to know how to properly involve it in a join. All the arguments would be strings, so I would have no idea if they were properly formatted or not until the database spit back a syntax error - which might not contain any useful information.

If I were generating SQL by hand though, I doubt it would have occurred to me to write such a function in the first place. I just would have written the SQL code necessary to do this for the calendar_event table and moved on.

If your database wrapper is going to do caching, queries have to be introspective so that the cache manager knows when a given chunk of SQL might invalidate your cache. Managing caching plus concurrency is hard enough with help from every layer of your system.

There are also some features which are directly supported by a complete object model for queries.

The feature I'm going to be working on tomorrow, a generic browser for tabular data, is another example of where having as much data access as possible happen through objects is helpful. The tabular data browser, or "TDB" as we call it, takes a query-like object as an argument, so you can page through complex queries. That object encapsulates both information about the objects being queried and the database itself, the TDB can both display the data appropriately and easily and quickly generate appropriate SQL. Without such an intermediary layer, the TDB would itself be a mess of string concatenations and quasi-generic SQL generation of its own.

Eventually we hope for Axiom to monitor queries, inserts, deletes and updates as they happen to provide a "live" query interface, which takes the same objects that SQL generation would, but provide an active view onto those objects as they appear, change, and disappear. If you're issuing UPDATEs and INSERTs to the database without going through an intermediary layer there is nowhere to catch this but triggers - and again I find a point of agreement with DHH, the database is not where the smarts belong.

There's a new release of Epsilon today.

As it seems to happen every release, we added some new features in the course of doing the release itself. This release's showcased new feature: setuphelper.autosetup().

A normal project's setup.py might look like this:

from distutils.core import setup
setup(name="MySpiffyProject", version="1.2.3",
    packages=['mypackage', 'otherpackage'],
    package_data=dict(
        mypackage=['data/static.png', 'data/static.html'],
        otherpackage=['templates/*.kid']))

Now, this is somewhat fragile and error-prone. You can easily forget code which is supposed to be installed. Each new release requires the release manager to scour the directory tree for new files which might be required.

Twisted has a fairly baroque setup.py which deals with this problem, among other things. However, Twisted's setup.py (and other projects' as well) are rarely designed for re-use. Twisted has, in fact, entirely given up on setup.py sdist for generating releases.

Enter Epsilon's autosetup() - that same setup.py with Epsilon would look like this:

from epsilon.setuphelper import autosetup
from myproject import version

autosetup(name="MySpiffyProject", version=version.short())

These helpers make release maintenance a lot easier. You no longer need to:

• update your setup.py with each release: just the project's version number.
• 
  
• manually identify what files are required for the project to run: if it's in a Python package directory, the assumption is that it's there for a good reason
• 
  
• update your setup.py when you add new files
• 
  
• regenerate Twisted plugins caches post-installation

I need to look at setuptools and see if these concepts can be re-applied. If we're lucky they'll be unnecessary there, but just a little bit of automation has gone a long way so far.