Alternate Titles:

I Have No Log File, Yet I Must Scream

or

So You Logged It, Now What?

or

This is why I don't want to have LOG_DEBUG in Twisted

Sometimes, when writing a program, you feel compelled to make the program emit some output which is peripheral to its operation.  The question is - who wants to know about that information?

Maybe you're debugging the program, and you insert a simple 'print' statement to get some information about it.  Maybe your program is a network server, and you are recording the fact that a message was received and processed.  Maybe you're maintaining an old library routine, and you want it to emit a message that points to a newer, better version of that routine which is now preferred.  Finally, regardless of what kind of program you're writing, maybe it has produced an error that a user or administrator will need to deal with, and you would like to show it to them.

This activity is referred to in several different contexts depending on how the messages are delivered, but it is most commonly known as "logging".  It is critical to the operation of many, many different kinds of programs.  Unfortunately, it is one of the most poorly-understood and poorly-implemented areas of software in general.  Software is a veritable cornucopia of poorly-understood and poorly-implemented ideas, so that's really saying something.  You can see some of the more hilarious and visible examples of developers getting this wrong in the "Pop-Up Potpourri" series on the Daily WTF.

It might seem odd that I lump together funny dialog boxes with "logging".  A dialog box is a little square on your screen; a log message is some text in a file somewhere.  But they are very much the same thing, and they fail in very much the same way.  Log files just do it less visibly.

The point that I hope to communicate here is that for every producer of information, there is a consumer.  When most programmers need to produce a "log message", however, they are thinking only of getting the information out of their program in some format, any format; not how that information is going to be used later.

When I say "most programmers", I most definitely include myself.  I'm probably guiltier than most. one of the reasons I'm writing about this in the first place is to work out some better approaches to the problem.

Consider this output from the "tomboy" desktop sticky-notes program on Ubuntu Hardy.  If I start it from the command-line, I see this:

[DEBUG]: NoteManager created with note path "/home/glyph/.tomboy".
[INFO]: Initializing Mono.Addins
[DEBUG]: AddinManager.OnAddinLoaded: Tomboy.Tomboy
[DEBUG]:            Name: Tomboy.Tomboy,0.10
[DEBUG]:     Description:
[DEBUG]:       Namespace: Tomboy
[DEBUG]:         Enabled: True
[DEBUG]:            File: /usr/lib/tomboy/Tomboy.exe
[DEBUG]: Updating note XML to newest format...

It goes on for several hundred more lines just at startup, and continues to produce messages as the program runs.  These messages are diligently classified into categories: DEBUG and INFO.  I'm sure they're useful to someone.  But why am I seeing them?  I just wanted to start a program to put some sticky notes on my desktop, and none of this information is useful to that task.

I have to imagine that pretty much all of these messages are useful only to Tomboy's developers.  But, worse than the fact that I see them is the fact that if something really interesting happened — I discovered a critical bug, let's say — all of that log output which is being splatted onto my screen is going nowhere.  It is a book written on water.  (Well, a book written on video memory, which is pretty much the same thing.)  Meanwhile, thanks to the bug-reporting facilities in Ubuntu, I'm sure that I could opt to give the Tomboy developers a huge ton of mostly useless information, like the contents of my registers at the time that it crashed.

Consider not just the placement of the messages (on my screen, where I certainly don't care about them) but their formatting.  Who is that elaborate right-justification of labels in the "DEBUG" output for, anyway?  It isn't for me, I don't want to see these messages in the first place.  I doubt it helps the developers, either; rather than just grepping for '[DEBUG]: File', now they need to put in a regular expression to collapse whitespace, or count the number of spaces that the justification happens to put in.  Presumably if this output is useful at all, it is useful in a search.

Text Formatting and the Inevitable Descent into Log-Level Hell

The right-aligned pretty-printing is a beautiful illustration of a very common anti-pattern in logging: trying to convey structured information by messing with a textual format.  A developer wanting to write a message indicating that there is a problem with the program, left with the extremely narrow confines of a logging API which just takes a string, will often do something like this:

log("*****")
log("THIS SHOULD NEVER HAPPEN!  HELP!!!")
log("*****")

Of course, this frantic wording doesn't help the output go anywhere but silently into a log file where it will be ignored.  But, perhaps if this is some server software, an administrator will notice this message and set up an alert that makes their blackberry buzz when they notice those particular words show up in the log file so they can ssh in and look for problems.

Then the developer gets chastised by his manager for his un-informative error message, and updates it to be something clearer:

log("Serious Error: phase inducers have been depolarized.  Contact engineering immediately.")

Of course this breaks the administrators' alerts, so after much discussion between programmers and admins, log levels are added so that admins only get alerts when something "really bad" happens, where "really bad" is an agreed upon flag:

log2(SERIOUS_ERROR, "phase inducers have been depolarized.  Contact engineering immediately.")

Okay.  Now we've got a log level so admins can tell when their pagers should go off.  Except, different developers have different ideas about what "serious" means.

log2(SERIOUS_ERROR, "OMG I lost my cat Mittens.  Where is my cat?")

Clearly this is an abuse of the new "severity" flag that was added, but the cat-engineering team thinks that loss of a cat is pretty serious, so we add a new thing, a log "system".

log3(SYSTEM_CATS, SERIOUS_ERROR, "OMG I lost my cat Mittens.  Where is my cat?")

Most logging systems stop in this general vicinity, but we still haven't solved the problem, which is that the log message has no structure and you can't tell what's going on without groveling around in a bunch of text files with regular expressions or manually reading each message.  Which cat was lost?  Which phase inducer was depolarized?  How do we get from a log message or alert to this information?  The 'log levels' solution to this problem is clearly untenable:

logRidiculous(SYSTEM_CATS, ALERT_IF_YOU_LIKE_CATS, O_RLY, YA_RLY, SERIOUS_BUSINESS, BUT_NOT_TOO_SERIOUS, CAT_LEVEL("Mittens"), "OMG I lost my cat Mittens.  Where is my cat?")

More importantly, if you're writing a library, you have a bunch of other problems.  This diagnostic information needs to be logged somewhere, but what if this library is being used on a user's desktop machine?  Some of these messages are relevant to them as well.  How do you tell the user who is using a GUI that a cat has been lost?  How do you show them the picture of Mittens so they will recognize her if they see her?

Everyone agrees that log messages need some "small amount" of information associated with them, but very few people can agree on what that information should be.  Even at the simplest layer, the idea of a "level", there are lots of open questions.  Is the "debug" level for a programmer trying to debug something on their test rig, or is it for administrators trying to debug something in production?  Should there be a difference between those two things?  How serious does a problem have to be before warranting a "critical" classification?

Once you're using logging code written by more than one programmer, or worse yet, more than one team, you're going to be facing this problem.

The Particular Problem of Libraries

This is, of course, my main interest, since this is where the rubber meets the road for Twisted.  Libraries need to communicate to several different audiences:

1. We need to tell developers using the library about the correct way to use the library at runtime.
   
2. We need to tell administrators of systems using the library about the status of the library and tasks they may need to perform to keep it functioning well.  (Clear your caches, restart the server, install a security update...)  We also need to provide administrators with information they can mine for statistics about how the library is performing; how many requests handled, where its resources are going, etc.
   
3. We need to notify users of applications using the library about things that the library is doing which may be relevant to them.  (A new message has arrived, a new printer is available... obviously this depends heavily on what the library does.)

Libraries, especially event-driven engines such as Twisted, libevent, and glib, have a particularly difficult time because they have to deal with all of these audiences simultaneously.  However, I think that any application or server which needs to do some kind of logging or user notification needs a subset of these features, so if any logging system could solve this problem, it could solve pretty much all logging problems.

Type of Information by Type of Audience

Developers, Developers, Developers

Many languages don't have a solution to developer communication at all.  Python has one — the warnings module — but it is in many ways inadequate.

The warnings module doesn't easily let you selectively see which libraries you want to see warnings for.  If I'm developing an application A using libraries M, N, and O, which themselves have dependencies on X, Y, and Z respectively, I don't want to see warnings that M caused in X or that O caused in Z; those are problems for the maintainers of M and O.

I am maintaining only A, so I want to see warnings caused by my application in M, N, and O.  I can try to filter specifically by module, but unfortunately the
only way of determining which library caused which issue is by directly
examining stack depth, which is unreliable at best and misleading at
worst.  Even if I could filter very accurately, it's hard to get a stand-alone report of warnings and deal with them as they're supposed to be.  Warnings show up to end-users as well, and to administrators looking at applications in production.  It's worth putting up with that to have at least some solution for communication with developers, but it would certainly be better if it didn't happen.

Finally, it's easy to generate a huge amount of warning noise (and, especially as of Python 2.6, many libraries do).  With that much noise and no reporting functionality it's hard to the warnings you care about.

A better solution for communicating for developers would be one which:

• allowed developers to declare somewhere what code they are working on and what code they are just using
• recorded relevant warnings to a log file which was optimized, perhaps with an associated tool, for locating and removing the sources of the warnings
  
• allowed end-users to easily communicate their warning data to developers without inundating them with irrelevant noise while using the application
  

Administrators

To communicate with administrators there is a huge variety of options, but many of them depend on a lot of ad-hoc hackery by the admins themselves, which means they are inconsistent and therefore there is little reusable technology or standardized APIs available.

Right now the gold standard for talking to admins seems to be just writing strings into a text file and hoping they have some facility to read it.

A better solution for communicating with administrators would be one which:

• preserved structured data in an analysis-friendly format, rather than formatting it in human readable messages. (For most UNIX admins, I imagine some kind of structured text would be best, so "grep" would still work but more advanced tools could also be brought to bear.  I'm not sure what the tools in the Windows world look like.  The "Event Viewer" looks like maybe it's a step in the right direction, but its UI is incredibly primitive.)
• provided easily-accessible hooks for dispatching different types of events to ad-hoc code to wire up to existing notification systems - without significantly altering the behavior of the system doing the logging, if the logging hooks were broken, as admin-written code tends to be a bit flaky
  
• included an enumerated list of events which administrators could inspect before they happened to run across them in log files

Although it's a crappy format in many ways, the Common Log Format for HTTP might serve as a good example.  Unfortunately it's too purpose-specific to extend to do more than what it already does, but lots of tools have been written to produce lots of interesting data from even that very simple standard.

Desktop Users

There are two popluar cases for communicating with end-users.  One case is that you're actually running a program on their desktop and you want to tell them something.  Another is that you've got some code running in a web application which wants to tell them something "out of bounds".

On the desktop, there are fairly standard "notification" APIs for popping up little bubbles.  On the web, there are emerging conventions for these notifications, like a bar that descends from the top of the page to mimic the firefox 'do you want to remember this password?' UI element.  A good example of this is Stack Overflow's notification banner.

Unfortunately both of these have a problem with scale and with timing.  If your application suddenly encounters a large number of errors, it will flood the user's screen.  If the user isn't present when a notification occurs (or navigates away), the bubbles or banners may disappear.

A better way to talk to end-users would be:

• for desktop applications, a mini-email interface, which records notifications in a scrolling list so that users can inspect notifications that occur while they're away.
  
• for web applications, a standard API so that multiple applications on a single site (or even, potentially, on different sites) can drop notifications into a queue which can be displayed appropriately.  (Since websites tend to have strong preferences to control their own design, an actual standard widget might not be possible, but it would be nice.)
• for both of these, a standard protocol which would enable notifications to be easily streamed to different computers or mobile devices without needing to reconfigure
  
• a specific classification of messages at the API level, saying, "I want to tell the end-user about this".  Messages about crashes, etc, should be displayed as an option to send information to the developer.  In the context of a web application this can be done automatically and silently; in a desktop app there would need to be some channel set up for sending that information.
  

Let's not forget that administrators are users too.  Everything that happens in a server's notification APIs should be able to be trivially filtered and redirected to administrator's desktop machine (or their phone) so they can immediately notice when something has gone wrong.

Appendix A: Optimization and Dynamic Instrumentation

This doesn't fit into the "figure out what you need to say and who you need to say it to" theme of the rest of the requirements here, but it is nevertheless important.  If you make heavy use of a logging system, especially one where you have lots of messages that are logged "just in case" and rarely displayed, you will quickly discover that it's consuming a lot of resources.

The work to calculate what goes into a log message shouldn't be done unless the message is actually going to be emitted.  Similarly, it should be easy to dynamically add and remove log events from particular methods or particular objects without modifying their code.  The best way to do this is to keep the logging entirely separate and add it at the level of methods and functions, rather than ad-hoc in the middle of application code.  Sometimes, of course, you want to emit log messages at a very particular point, but in general it should be easy to add instrumentation to a method without modifying its code, to avoid cluttering up application logic with lots of "just in case" debug messages.

What Can Do This?

I'm not aware of any logging system that can already do these things.  We'd have to write a new one.  This essay was largely composed due to my desire to understand what I thought a "good" logging system would do.  It might be too ambitious.

There are a few fundamental units which are missing from most logging systems.  While lots of logging systems have various ways to indicate subtle and nuanced levels of urgency, few have a way to indicate who a message might be relevant to.  Logging systems also generally don't have a good way to associate structured information along with a log message.

Twisted's logging mechanism does have a free-form dictionary associated with each message, but that's not much help unless you impose your own structure on it, which means you have to build all this infrastructure anyway.  It is, at least, possible to treat the Twisted system as a kernel which this could be based upon.

In order to produce an enumeration of events before the events are actually logged, this API will require pre-declaration of log messages.  This might be too much of a burden, since sometimes you want to just stick a log message into the middle of some code so that you can see if it happens.  So, in practice, it's more likely that pre-declaration will need to be optional and you'll need to be able to associate ad-hoc data and have it still be persisted along with your message.

There will need to be some way for communicating both the structured data of an event and the human-readable text associated with that event — preferably in a way which can be internationalized.

There's also a bunch of UI, web, and protocol standardization work that would need to be done.  Luckly, that's independent of the actual log machinery; if it already existed it would be a trivial matter to hook it up.  In the meanwhile, something that did all of this but just used existing facilities, like the desktop notification spec, status icons and email, would still be immensely useful.

I've kicked off our 2009 sponsorship drive with a post on the Twisted Matrix Labs blog.

Please share and enjoy.

This is posted mainly for my own benefit, so that I won't have to re-remember the command-line options to make this work for the nth time in a row, but some of you may enjoy it:

    easy_install --prefix ~/.local/ --site-dirs ~/.local/lib/python2.5/site-packages your_package_here

Now that I've gone through and understood why that's necessary, I think I might be able to fix it in Combinator one day...

I have recently been working on some applications that make use of the features provided by notification-daemon.  I installed Jaunty to check out how the much-vaunted new notifications framework works and how it would affect these applications.  I was aware that it sacrificed some features, but I wasn't worried.  I am generally a fan of the GNOME philosophy of dropping functionality and configurability when it doesn't really serve the user.  I also appreciated the new, more minimal and sleek-looking graphic design of the notifications.

I feel like I need to preface my reactions with an apology.  I really like Ubuntu.  In many ways Jaunty looks really slick, and I'm generally enthusiastic to upgrade.

But this notifications stuff?  Wow.  What a disaster.

First, a little background.  I recently started working on two different applications which made heavy use of the notification API. While I originally thought I'd need to implement some of the notification features that I wanted on my own, I was pleasantly surprised to discover that notification-daemon provided almost all of them.

I want to present the user with a time-critical notification, one which didn't grab the focus and interrupt their work.  I want to show the user how much time they had to respond, and provide a few different options for responding to certain notifications.  In one application, I'm implementing a sort of dead-man's switch, where failing to respond within the time limit also counts as a negative response.  I also want to emphasize certain notifications, and provide hyperlinks to their web-based origin so the user can jump straight into the application at the appropriate point if a notification is interesting.

Some of the notifications I want to generate are notifications of events, some are indications of a change in status of my application; so sometimes I want to point at a particular status icon and sometimes I wanted to just drop the notification into a queue - hopefully a global queue which would intermix with other notifications.

I was assuming that I'd need to implement some of these features myself, but to my pleasant surprise notification-daemon handled every single use-case, more or less exactly how I envisioned it working.  I was thrilled.  As part of searching around for "notification" stuff, I learned that Jaunty will have some newer, even cooler notifications stuff, and I was really excited.  Unfortunately, Notify-OSD, the new Ubuntu-specific notification daemon, drops nearly all of these features.  So, I'm back to square one.

Here are some of the specific things which bothered me about it.

1. Applications which emit a notification that prompts for an action ­— something which they only would have done if they explicitly wanted to avoid grabbing focus, since popping up a dialog box is easy enough — will have a modal dialog box pop up and grab the user's focus while they're working.
   
2. Timeouts are no longer honored.  It so happens that in my application I have an operation which takes 2 minutes to time out; I would really like my notification to stay on-screen for this entire time.  I can still do this with notify-osd, but in order to do so I have to watch for the "closed" event and constantly create new notifications.  A smoothly animating timer was a much nicer interface than a sequence of bubbles saying "In 30 seconds I will time out.  In 25 seconds I will time out.  In 20 seconds I will time out."  Yes, I realize that the Ubuntu desktop team will say that I should do something different in this situation, but they're not designing my application and I don't like the options they've suggested.  This would be much less of a problem if the provided notification timeout weren't so distressingly fast.  As a user, by the time I've realized that a notification has popped up and taken the time to focus my eyes on it, it disappears halfway through my reading its message.  I am a very fast reader, and I'm pretty sure that there are a lot of people who are never really going to notice any notify-osd bubbles; they're so fleeting, they're just visual noise.
   
3. Markup is now silently ignored.  I can't emphasize portions of a notification with a larger font, or provide a hyperlink to the origin of the notification.  Similarly, since actions have been broken, I can't provide an action to jump to what caused the notification.  Notifications have thus become disconnected UI element which tell user something while providing them absolutely no tools to deal with it or respond to it.  For example, when Pidgin tells me that a user has signed on, I can no longer interact with the notification to say "yes, I would like to talk to that person".  I have to switch windows to the buddy list, locate the person who just signed on, and click on their name, rather than just clicking on the notification itself.  Or, I have to click on the tiny notification-daemon status icon that's a hard target to hit with my mouse, rather than a big friendly button.
   
4. Notifications are now displayed in the upper-right-hand corner of the screen (where important chrome, like close boxes, search boxes, toolbars, and menus frequently reside) rather than in the lower-right, where less important application features are traditionally located.  Granted, this is a damned-if-you-do-damned-if-you-don't situation, because some applications put important action buttons in the lower right, but I have been learning where to position my windows to avoid that area of the screen for years, and now I have to learn new habits.
   
5. Notifications can no longer be positioned on screen, relative to either widgets in windows or status icons.  This removes the ability for an application to use notifications to draw attention to a particular area of the screen.  Instead, users must make some connection between the notification bubble and the status icon themselves, perhaps by identifying some common graphical element.  If you're already using the "icon" area of the notification to display a picture (such as a person's portrait) it's a bit cramped to also show a copy of the status icon, especially given that the icon will now be squished for you so it's hard to get a pixel-accurate rendering of the status icon anyway.
6. Some of these problems are nominally addressed by the new "indicator applet" facility in Jaunty, but...
   
   1. The indicator applet and libindicate library appear to be almost completely undocumented; the "reference manual" looks like it was an auto-generated stub.  The automatically generated API documentation isn't hosted online anywhere.
   2. The python bindings for libindicate are similarly undocumented, and they aren't packaged anywhere, not even a PPA.  They also use autoconf, rather than distutils, for installation, so their build process doesn't produce a usable extension module and thus they resist installation anywhere but in /usr.
      
   3. I tried to read what passes for documentation — the patch to Pidgin's libnotify plugin that switches it to use libindicate for some things.  I tried it out because I wanted to see if maybe the indicator applet could address some of my concerns, but my misuse of the API caused the indicator applet to instantly segfault.
   4. From what I can tell, you can't just provide an icon and some text, you need to actually create a .desktop file, which means that packaging applications which want to use the indicator applet automatically gets two additional layers of complexity: first, you need to create a .desktop entry, and second, you need to figure out a way to have your application include it during installation.
      

It's interesting to read the version history for Growl, the OS X notification tool which so clearly provided the visual inspiration for Notify-OSD, and notice that many of the features now being removed (application level positioning, close buttons on notifications) are features which were added to later versions of Growl.

The "notification design guidelines" provide some very vague suggestions for ad-hoc mechanisms to work around these regressions.  These suggestions are unhelpful.  I want an API that I can call, not a picture of a window I need to re-create myself.  If the desktop team wants to change the look of my application in the future, I don't want them to submit a giant pile of patches against it.

Not only are the suggestions not a library, they don't include sample code, either.  How do I actually create an alert box which doesn't take focus, doesn't include any window manager controls, but stays above other windows?  Describing it this way is an invitation for applications to be inconsistent.  My interpretation of this specification will inevitably be different from other application authors.  For example, the specification doesn't say anything about compositing, but the window in the screenshot clearly appears to be alpha blended with the background.  It also doesn't say anything about WM controls, but some pictures have minimize/close buttons and some don't.  Some applications will have alpha blending for their notifications, some won't.  And since there's no library here, there's no reasonable way to consistently control the behavior of many applications.

With these features in libnotify, we have a single uniform queue for users' attention.  A single point of control which might be adjusted, bugfixed, and tweaked across the desktop as a whole, without writing tons of patches for individual applications.  Notify-OSD even takes advantage of that point of control.  But the suggestions for many of these features is to take control out of a single easily-managed client/server protocol and push it into a bunch of ad-hoc application-specific widgetry.

To add insult to injury, the one place that I do get hard examples of how to do things, on the notification development guidelines page, the Python code samples have yet to be written.  This is a minor nit, as I can definitely figure out what's going on from the C# code, but it's endemic of the same systemic problem with the Linux desktop ecosystem that brought us this half-baked replacement for notification-daemon.  Jamie Zawinski identified this problem as the "Cascade of Attention-Deficit Teenagers", but Canonical demonstrates that you can create this same problem with a medium-size company that employs highly competent, adult engineers.

Notification-daemon isn't perfect.  It could clearly stand to be improved, especially in the face of notification spam.  So please, improve it!  Or, at worst, if upstream is not cooperative, fork it.  I'm pretty sure that the solution to the rate limiting problem is not "then... what?".  Notify-OSD cuts the gordian knot of notify-spam by only letting you see one thing at a time, but that has its own problems.

Now, to be fair, all these regressions haven't really cost me any work.  I want my applications to be cross-platform, so I was going to have to implement most of this functionality for Windows anyway, using animating borderless windows.  Now I'm just going to be using my own notification widgetry on Ubuntu as well, rather than elegantly integrating with the platform and providing all of my notification interaction through a familiar UI.  But I'm sad that the superior notification infrastructure on Linux in general and Ubuntu specifically is no longer something that makes my application easier to write first.

So, beyond this one little screed, I'm really not going to complain too much.  I'll implement some of my own ideas for notification, try to come up with some way to be friendly to Notify-OSD in the meanwhile, and I'll still eventually upgrade all of my computers to jaunty and enjoy the other eye-candy and performance improvements.  This is not too terrible of a price to pay, and I do keep it in perspective.  I also understand that I'm not the guy who has to make the hard decisions for what goes into Ubuntu or GNOME or whatever.  I understand that sometimes, in order to make an omelette, you have to kill a few people.

But, if a decision maker for Ubuntu were to care about my entirely irrelevant opinion, as both an application developer and heavy user of notification systems, I would say this to them:

You guys have done some great work on Notify-OSD.  It's a worthy prototype.  In many ways it is better than notification-daemon: it looks nicer, it makes notifications between applications more consistent.  I can appreciate the uncompromising vision you have for cleaning up the sometimes confusing pile of notifications that users see.

You should package Notify-OSD in Jaunty, so that people start using it.  Start updating applications to honor the capabilities that it provides.  But please, don't make it the default in the first release where it's included, and yes, include a preference for the period of transition.  Write some libraries to support the other use-cases which Notify-OSD right now ignores.  Document and stabilize the indicator applet.  Package the Python bindings, please.  Make it not crash when applications abuse it.

Regardless of this new notification system's unpolished state, I'm sure many users will update and start experimenting with Notify-OSD, much as many started with Compiz for years before it became the default window manager.  Most users can keep using the regular notification bubbles until Karmic, though.  When Karmic comes along, you'll have had the time you need to finish the documentation and provide application developers better alternatives to a good notification API before yanking the carpet out from under them.

If this advice is ignored, as I'm almost sure it will be, it won't bother me - notification is hardly the most important API that an OS provides.  The thing that I really hope someone will take away from this is the general theme that platforms should evolve experimental features slowly, and you should always have a well-documented, better alternative ready before you remove something.  Notify-OSD removes a half-dozen features and informally, halfheartedly gestures at some ways you can make your window pop up to address what some of those features used to do.  That's fine, as a scrappy new competitor to notification-daemon, but not as a core part of a major platform.

My real hope is not specifically that Notify-OSD will actually be pulled.  Of course I'd be happy if it were, but Jaunty going to be released in just a few days.  Again, I feel like I need to qualify these statements: if I'd really wanted to impact decisions like this I should really be regularly using beta releases.  Not to mention the fact that if I'd actually finished these hypothetical applications I'm thinking about, I'm sure my voice would carry more weight.

My real hope is that you, gentle reader, will take this message, and the next time you are contemplating boiling your favorite ocean, you'll stop and reflect.  Break down the changes you are planning on into individual, incremental improvements, rather than sweeping, break-everything lateral movement.  Make radical improvements, but make them behind the stable facade of a system which is only lifted when the radical improvement is clearly both radical and an improvement.

I tried to write a conference wrap-up, but there's just too much.  Besides, Ted Leung already wrote a better one than I could.  (I'm pleased to note that the first talk he mentions is the only Twisted talk at the conference: "Twisted AMQP and Thrift".  Go Esteve!)

I am once again impressed by the conference organizers.  Every year it seems like the conference gets better.  This year, it seemed like nothing went wrong - which wouldn't be remarkable if I didn't know just how difficult it is to create that impression.  I registered for the conference online, reserved my hotel room through the housing committee, and when I showed up everything was ready.  The food was good, all of the staff were helpful and efficient.  There was plenty of room for everything, even though the Open Space board was completely packed.

The video was particularly awesome.  I didn't get to see many talks, as I was moving from one conversation to another in the hallways, but I was impressed to discover that some of the talks I was hearing about during the conference were already online so I could catch up on the buzz.  The camera work and editing are really good.  I particularly liked that they often used picture-in-picture to show both the presenter and their slide.  But, you don't have to take my word for it; check them out now on pycon.blip.tv.

My personal experience of the conference is of course defined by hallway chats, open spaces and sprints.  This year we had a great Twisted open space, followed by a great Twisted Web open space the next day.  Thank you to everyone who came.

These sessions helped to reinforce for me the need to repeat this frequently for our users: if Twisted is doing something which confuses you but seems wrong, please go ahead and file a bug on twistedmatrix.com.  If you're wrong, and Twisted is working properly, there's still a bug, it's just a bug in the documentation.  If the documentation were perfect presumably you would understand why it's doing what it's doing.  Even if we ultimately decide that the documentation is sufficient, by adding a bug in a tracker, you've told Google that people with your question should find the document we refer to in our comments, so there's still value.

The sprints also went really smoothly this year.  Power and wifi were in abundance, so we could all just get to work.  I'll tell you a secret, though: I never plan to get much work done at the PyCon sprint, especially with monthly Twisted sprints here in Boston.  The value in this nation-wide gathering is in helping new users get up to speed with hacking on Twisted, and in running around talking to people sprinting on other projects, getting them to integrate with Twisted.  This year I had a pretty focused message that I wanted to get out: Twisted is a WSGI container, and it's one you can invoke from the command line with "twistd web --wsgi=your.application.here".  In other words, even if you prefer to write synchronous, blocking code, you can still use Twisted to run your web application, and you don't need to write any additional code to do it.  As a result of this communication, David Reid fixed a few minor bugs in our WSGI container and Twisted trunk now runs Django as well as Pinax.  I hope that this will drive even more adoption of Twisted in the Python world.

Also, it seems like for the last few years I've gotten started thinking about PyCon talks too late, and by the time the deadline rolls around I've got nothing.  In fact it seems like this has happened to most Twisted devs over the last few years.  This year I plan to get started right now.  I hope you'll join me, so that we can have a "Twisted comeback tour" at PyCon 2010.

One last thing: at least three people approached me at various points during the conference to ask me about using Mantissa and Axiom, and I didn't have a chance to catch up with any of them.  I feel bad about this.  If you tried to talk to me about using some Divmod technology at pycon, and still want to talk about it, please feel free to send me a personal email; we can set up a Skype session or something.