Post PyCon Wrap-Up

Saturday April 02, 2005
A bit late for this sort of post, I know, but (as my previous post indicates) I spent the week following PyCon convalescing. I'm feeling a bit better now, so I figure I should share.

Things started off at the sprints, during which Twisted 2.0 was released. Chris Armstrong, the release manager, was in Australia, but it was well timed anyway. We had a nice big sign and I think that the whole sprints qualified as the "release party". The Twisted table this year was huge - we probably could have had an entire room to ourselves and still been short on space. The final day of the sprints people were literally having to stand around for want of chairs. It's fun to be at the center of a whirlwind like that.

Early on James Knight and I sat down with Jim Fulton and talked about Zope Interfaces. I wish I could say that something earth-shattering came out of that, but mostly we just agreed that they have been working wonderfully for Twisted so far and that we will continue to use them.

There were some new faces this year: I got to meet Thomas Vander Stichele from Fluendo, Abe Fettig of Hep fame, Patrick O'Brien from the Twisted list, and Timothy Fitz from IRC - not to mention Phil Frost, the only member of the Divmod team I'd never physically seen before.

During the conference, 5 Divmod people gave presentations. In the order they appeared:
  1. An overview of LivePage, by Donovan Preston
  2. A talk in Open Space about using the Xen virtual machine monitor to facilitate secure, "agoric" restricted execution for Python programs, by Allen Short.
  3. Xapian indexing & Xapwrap, by Michael Salib.
  4. A demonstration in Open Space of the Quotient application and the Divmod service, by Amir Bakhtiar.
  5. A description of the in-development peer-to-peer application protocol, Q2Q, by me.
All five talks were awesome, of course. In both Mike and Donovan's talks I learned surprisingly pleasant things about our own code, which I guess is an indication that we should go to more conferences...

I spent a good deal of the sprints, and the conference, preparing for my talk and writing code to test the ideas in it, so I missed a lot of talks that I wanted to see.

The nice thing about being at a conference like PyCon though, is that if a talk is good you'll hear it repeated 3 or 4 times in the halls as people are discussing it. I heard a few very, very interesting talks in this manner either from their authors or from attendees of the talks. "How to Build an Air Traffic Control System" involved Twisted, and it really blew my mind how good a fit it was. "Yarn: Working with Messages in Diverse Formats and Protocols" was some real food for thought, and I've been looking at yarn quite a bit since then - I really like it. The Chandler demo talk impressed my co-workers enough that we are now anxiously awating a release we can target as our primary client platform. Finally, of course, I've been hearing about Itamar's "Fast Networking with Python" talk for the last few months and I can't wait until the optimizations he's working on are just part of the default Twisted reactor.

There was a lot to assimilate in only a few short days - and of course I've left out days' worth of side-conversations with old friends and acquaintances from previous conferences about a huge variety of Python, Twisted, and Divmod-related topics.

Unfortunately the Divmod team came to the conference after a very hectic month where most of our preparation time was eaten up by more pressing concerns1, so we weren't able to update the divmod.org website in time for the conference, which was disappointing. Also, the whole conference was a rollercoaster ride for me, since I was still working on proving the concept that my talk was supposed to be about.

Still, we managed to sign up a bunch of new customers2, and there were a lot of people randomly walking up to me throughout the conference and saying how they thought Divmod was cool. Hearing that from people who aren't involved in the daily struggle is energizing, and worth the cost of admission by itself, for me.

As great as all that stuff was, I wouldn't be accurately summarizing my experience of this conference for myself if I didn't talk at least a bit about Q2Q. I explained it about 12 different times, slightly differently each time, to other programmers at the sprints, then I paired with a half dozen other people throughout the rest of the week.

Since there is no real documentation yet (still working on it!), I'll give a brief overview.

On today's internet, when you want to connect to a service, you have to know a host name, and a port number (or at least an implied port number). Q2Q is an attempt to take the same underlying model of TCP - end-to-end streams of bytes with the physical transport medium abstracted away - but identify the endpoints in a more human readable way, and remove some of the transport-layer obstacles that have crept into TCP over the years. Rather than connecting to your-host.com on port 80, you connect to "your-user@big-host.com" over the named protocol "http". Also, you specify who you are connecting as, which is a persistent, reverseable identifier, unlike the IP address and port number associated with a TCP connection. Additionally, since Q2Q introduces a new level of indirection and a new lookup layer, it also adds some automatic NAT-traversal logic, which means that it will let you connect to computers that are behind firewalls which normally disallow incoming connections - without creating a security problem, because Q2Q connections have to be explicitly requested by first creating (and maintaining) an outgoing connection.

There are also some interesting security properties of connections established with Q2Q, but those are more difficult to explain and more in development, so I'll leave it brief - you're a lot surer of where the connection is coming from over Q2Q than you would be over regular TCP, and it's got some built-in features to help prevent problems of spamming. There is a SSL-secured trust-web automatically expanded by every new connection, which can be used to detect spoofing attempts and trace the origin of malicious traffic.

Also, by using that trust-web established between servers, Q2Q can be used as a distributed authentication mechanism for the web, or as the kids are calling it these days, a "federated identity" system.

I am personally very excited about working on Q2Q. I feel like in these days where everyone has 50 different accounts on 50 different websites, and there's no simple way to even transfer a file between two computers that will reliably work without always wasting huge amounts of some passive third party's bandwidth, let alone reliably hook up a private peer-to-peer chat room, something that offers these features is really necessary.

A few people have pointed out that the SVN URL that I gave at the end of my talk was wrong. Since it hasn't yet been merged into Quotient's trunk (though it will be any day now - I'll post an update when it is!) for the time being you can see it at:
svn://divmod.org/svn/Quotient/branches/glyph/pycon-q2q/

Although the demo at the end of my talk was a disaster (just a terminal window with a traceback in it, basically), some good did come out of the late nature of its development - I got to work with a bunch of people on q2q that I otherwise wouldn't have. Since I wasn't going to blame the broken demo on them, I didn't mention them at the end of my talk, but now that things are working again (albeit in a branch), I want to send a really big thank you to the following people who paired with me on Q2Q during the conference:
  • Pavel Pergamenshchik
  • Alex Levy
  • Moshe Zadka
  • Brian Warner
Without their help I wouldn't have had a talk at all, nor any code to show now.

To sum up - PyCon 2005 DC was fantastic, as in previous years. It still makes the #1 spot on my conference calendar. I am sad to see Steve Holden go as the organizer, since he really rescued the Python conference scene. IPC11 was starting to feel a little pathetic, and the Python track at OSCon isn't really a conference of its own. Still, I think that AMK is going to do a great job next year, and it looks like attendance is going to keep going up and up.

It's a great time to be a part of the Python universe, the Twisted cabal and the Divmod team. I'm already looking forward to next year's conference.


1: Divmod is raising investment capital right now. We've got a fair amount of interest, but more is always welcome - especially from the sort of people that would read this blog - investors committed to open source, privacy, security and generally not being an evil corporation. If you, or someone you know, is an investor like that, drop me a line.
2: If you're interested in subscribing to Divmod, contact our marketing & support "department" and let us know you'd like a ticket. We're trying to grow our user base by a few people each week, to gradually scale our server up to more and more load.