exarkun for president

Jean-Paul Calderone is an amazing hacker.

So, here's the setup.  We're working on this application that runs in the Adobe AIR runtime.  We implemented the AMP protocol for client-server communication.  Superficially, it worked great; limited tests gave us good results.

Then, we started throwing some real data at it.  And it choked.  The runtime would terminate its client socket silently: it would stop delivering data to application code, send a TCP FIN to the server, and not even deliver an event indicating that the socket had gone away.  Nothing.  Nowhere to set a breakpoint, nothing to debug.

The Python implementation of this protocol worked fine; everything got delivered.  The connection was not dropped unless we told it to drop.

I spent all night poring over protocol dumps, trying to figure out what was going wrong.  There were slight differences in where in the data stream it was dying - but for some reason, always cleanly, on a message boundary.

So, I come into the office and I fire up the program and show JP.  We get a tcpdump, and he looks at the output.  He squints at it for a few minutes and says:
"Huh.  It died on message 64.  That's interesting...
Oh wait, that's hex.  What's 64 in hex?  100.
... thoughtful pause ...
Maybe the garbage collector is buggy?"
He was right.  The bug was in the garbage collector.  AIR apparently doesn't think sockets (and apparently, other stuff, like animations) are "real" things that should keep strong references to the things they are feeding events to - so, sometimes they just crap out and get garbage collected, and silently stop delivering events.

The Ultimate Ubuntu / PulseAudio Guide

After writing about how great Hardy was, I suddenly started experiencing mysterious crashes of various applications on my desktop.  That'll teach me to say anything nice about an operating system!  Firefox, Pidgin, Deskbar, and even the Gnome Panel would periodically freeze up.  It seemed particularly exacerbated by running applications in the Flash and AIR runtimes — which was pretty bad news, since I am supposed to be writing some code in Flex!

Eventually, I caught one of these programs in the act, and traced the buggy behavior down to the PulseAudio daemon.

Apparently this is a common problem, with a bewildering array of elaborate and perplexing "solutions".  I spent hours trying to figure out some kind of combination of ALSA and Pulse tools that would get me out of the increasingly crashy situation on my desktop.  Oddly, nobody mentions the most obvious solution:

"Turn it off."

While I can appreciate the benefits that Pulse will (eventually) bring, it seems that it doesn't actually do anything for me today.  The main idea is that Pulse will allow me to play and record multiple sounds simultaneously.  However, the out-of-the-box Ubuntu ALSA experience allows me to do this with my current sound card.  I don't pretend to understand why — maybe my card supports multiple channels, maybe things are using dmix somehow — but there is no problem to solve.  Running the pulseaudio daemon actually has the opposite effect: sounds from multiple programs stop working.

I wanted to remove the pulseaudio package, but it turns out that would remove the ubuntu-desktop package as well, which means everything will probably be broken (certainly when I do my next major upgrade).  So, here's an alternate and non-obvious way to turn it off, without any fancy configuration:
  1. Go to the "System → Preferences → Sound" in your menu.
    http://www.twistedmatrix.com/users/glyph/images/content/blogowebs/sound-menu-item.png
  2. Make sure none of the devices say "Pulse Audio"; they should all say "Autodetect", "ALSA", or something more specific.
    http://www.twistedmatrix.com/users/glyph/images/content/blogowebs/nothing-says-pulse.png
  3. Click on the "Sounds" tab, and make sure that the "Enable software sound mixing (ESD)" checkbox is un-checked:
    http://www.twistedmatrix.com/users/glyph/images/content/blogowebs/quote-esd-unquote-disabled.png
  4. Log out, and let it trouble you no more.
I got a little chuckle out of discovering the "ESD" footnote, but it turns out that really means "PulseAudio" these days.

Better luck next time to the Ubuntu audio team!  I look forward to using all these cool pulseaudio gadgets next time.  I hope skype, flash, air, and the dozen other proprietary-and-wrong audio apps out there get fixed before Intrepid hits the shelves.

Hack, the Hardy Heron Sings: Glory to this Linux thing

It was time.

After running Gutsy Gibbon on five of my computers for almost exactly a year — nearly six months past its "best used by" date, and about seven months after I would normally have started upgrading — I have finally upgraded all of my computers which have displays.  Alastor (workstation), Illidan (desktop), Suijin (media center), Kazekage (backup), and Nhuvasarim (laptop) are all now happily flying the 8.04.1 banner.

You may have heard that this particular upgrade was somewhat rocky.  I have definitely seen a few of my friends lose hours of their lives that they'll never get back with 7.10 → 8.04 upgrades.  However, I'm happy to say that after paying my dues as an early adopter of inadvisably new kernels, packages, and libraries for ten years, my decision to lag this time has paid off.  On all five highly varied systems, the upgrade went off nearly without a hitch.

Nearly.

So, in the interests of saving those of you who are even later to the party than I am some trouble, I'll enumerate the few difficulties I did encounter.

The main thing that kept me from upgrading for so long is VMWare Server — or, as I sometimes like to call it while I'm trying to get it to work, VMWare CENSORED Server. There are packages for it that work beautifully in 7.10, but none in 8.04.1.  Considering that I have it installed on pretty much every computer that I use, that was a pretty big problem.  To make matters worse, not only were the nice, packaged ubuntu installations no longer available, some library incompatibility in Hardy made the custom-installer tarballs that VMware releases broken as well — and by "broken" I mean "would crash before opening a window".  But, there is now hope!  The recently-released 1.0.7 build of VMWare Server for Linux works like a charm on Hardy, with only two tweaks.
  1. If you're upgrading from Gutsy, and uninstalled your vmware-server package in preparation for installing the 1.0.7 tarball which actually works, you'll notice that the new tarball refuses to install.  That is because your /etc/vmware directory now refers to a removed installation.  You will need to back it up, since it contains your (ugh) "serial number" (seriously is this 1997 or what), and move it elsewhere.  Don't do a --purge removal to remove /etc/vmware, since that will destroy your virtual machines.  When the installer prompts you for a serial number, have a look in /your/old/etc/vmware/license.vs.1.0-00; it's helpfully left in plain text.
  2. You will need to sudo rm -fr /usr/lib/vmware/lib/libgcc_s.so.1 in order to delete a library which conflicts with something in Ubuntu.  A few months ago, there was a great deal more of this type of surgery, but thankfully this one step should be all, now.
So, once the finicky VMWare was up and running, it was time to resolve some testing issues.  Divmod's tests have a dependency on the now apparently unmaintained "pyxml" package.  Hardy has a package for pyxml, and it installs the code, but it does it in such a way that you can't import it.  I'm not sure why that's useful, but whatever.  Luckily, Combinator makes it so that "python setup.py install" works without being root, so I downloaded the ancient and whithered archive, which is now hidden for some reason, and installed it into my home directory.  Then I tried to forget about it.

I already had to make a tarball of an old, hacked up version of _PyLucene.so and friends to get the tests to run under Gutsy, so I didn't have to touch that to keep it working.  I guess I should make those files available somewhere, but I always feel like if I am going to put any kind of effort into making Divmod code easy to work with it will just be in ripping out PyLucene entirely, since that's tantalizingly close.

And... that's it.  Everything still works, and despite the occasional, although increasingly distant dread that random kernel bugs will attack me, number of things have been improved.  So, that's enough with the workarounds and hacks.  Let's get to the good stuff!
  • Firefox 3 is definitely faster.
  • The awesome bar is awesome.
  • Fullscreen flash movies are actually fullscreen, and don't reliably crash the browser.
  • The "typing" notification in Pidgin is a lot easier to spot and works a lot better in the flow of a conversation.
  • Deskbar can stick to the panel again, finally!
  • Compiz is generally less buggy.  I couldn't figure out how to file these as bugs in launchpad because I could never reproduce them reliably, but at least one of them would happen by the time X had been running for an hour and they're not happening any more...
    • The screen would sometimes freeze and compiz would crash while rotating the desktop cube.
    • Sometimes changing the title of a window rapidly (such as when typing commands in a bash prompt where the title is tracking commands or switching Emacs buffers rapidly where the window title is tracking the buffer name) would cause the entire window frame to appear unfocused until the title changed again.
    • Running games — WoW in Cedega, and OTRSPOD natively — under Compiz is much faster.  I can maximize WoW and it doesn't crash.  (Fullscreening OTRSPOD still seems to have a pretty vicious problem, but I can play it in as large a windowed resolution as I like.)
    • Sometimes window stacking Z-order would be messed up after invoking the shift switcher or application switcher; combined with the focused window frame suddenly becoming unfocused, sometimes I couldn't tell which window was actually on top or actually had the keyboard focus.
    • Okay, this wasn't exactly intermittent: the "Scale Window Title Filter" plugin would always crash after a few tries.  Now it doesn't.
  • The hardware temperature sensors can now tell me how hot my GPU is!  I'd be a little more excited about that if it weren't really, really hot:
  • On my laptop, suspend works when I close the lid.  By that, I also mean resume works when I open the lid.  This hasn't happened for me since Edgy, I don't think.  I can rest easy knowing that they're planning to break it again in intrepid, but maybe I won't upgrade to that right away either.
  • There are now free drivers available for my wifi card.  This isn't a huge win, since the ndiswrapper drivers still seem to have an edge in dealing with poor signal, but it's nice since the free drivers seem to get better throughput when the signal isn't terrible; and now, I have the option.
  • Although I still have a bit of work left to get it set up, two finger scrolling on the macbook trackpad now works.
Of course, the biggest good thing about Hardy is that it's the first major Ubuntu upgrade that hasn't really broken anything unexpected, nor taken any serious investment of effort for me.  The upgrades themselves, all together, took a lot less time than writing this post did.  My home machine upgraded while I entertained myself with "機動戦士ガンダム00", my work machine upgraded while I was occupied talking to JP about some unpleasant business, and my laptop pretty much just sat in a corner churning away while I was working.

Anyway, thanks to the Ubuntu distro team for making this a good one.  I'm certainly enjoying it!

-glyph

P.S. Sorry if this post seems kind of frivolous, but I have a huge backlog of writing to do and I needed to start with something simple.

P.P.S. I don't like to talk about personal stuff to much but last month I got married.  It was cool.

P.P.P.S. If you knew why the image above was appropriate to the theme of "hardy heron", congratulations, you are as much of a dork as I am.

Update: ScribeFire has a neat feature where dragging and dropping photos will automatically upload them to your blog's API.  I used it, and I was pretty impressed. 
ScribeFire uploads the images just fine, and gets correct URLs to use from the upload, and inserts the appropriate <img> tags into the content of the post.  However, Google's image hosting (picasa) has a bug in it.  It checks the referer to see if you are coming from a subdomain of a google "property" — google.com, blogspot.com, blogger.com et. al., and since my blog is hosted on a custom domain, it doesn't count as a "google property" despite the fact that it's on their servers and they know about it.  So, sorry if that last P.P.P.S. was kind of nonsensical without any pictures acutally visible in the body of the post!  I hope they're fixed now.  (How is it that I keep hitting these crazy problems?  It's not like I'm doing anything interesting, I just want to include indentation and images in my writing.)

Don't Call It Blogging

Despite my own impeccable credentials as an elite cyber-hacker, I am friends with a number of people who are bewildered by the profusion of different technologies that the internet now affords us to interact.  I recently had a conversation where someone was just confused by the whole "blogging" thing.  Why do people blagoblog on the intertron?  What is the point?  I'm a prolific "blogger" myself, I guess, but I found myself sympathizing as I tried to explain.

I'm a huge fan of the activity of blogging, but I have never liked the word, "blogging".  I never really understood why until I was attempting to explain what it's all about.

For thousands of years — well, okay I don't have any citations of exactly how long, due to the evolution of English as a language, but, for a really long time — we've had one word for the activity of "blogging".  We called it writing.  That's all you're doing when you're blogging.

If we were to describe the activity of a Sumerian scribe pressing symbols into soft clay, we'd say they were writing on that clay.  An ancient Egyptian putting words onto a sheet of papyrus: they are writing.  Similarly, we don't typically have separate words for "scrolling", "codexing", "booking", "newspapering", "magazining", and so on.  Each new technology for moving writing around didn't need a new verb.  So why has "blogging" gotten one?

I think there is a good reason this term exists, but that reason doesn't justify the term, it provides a warning, and a reason to try to actively resist the term and just say "writing".  The web is a more radical and democratizing shift in publishing technology than any of the ones which preceded it, so publishing on the web (especially automated publishing, as on a blog) affords a feedback cycle where the author and the audience are effectively peers.  In fact, the nature of the terms "author" and "audience" has changed; formerly a description of social classes, the people who produce and the people who consume, they have been re-framed as roles within an individual conversation.  You might be the audience when you're reading someone else's blog, but ten minutes later you can easily reverse that relationship with that author as you're writing your own.  This extremely rapid cycle has given a wholly new quality to the style of many blogs, unseen in any prior form of written media.

So, why resist the term "blogging"?  It confuses the possibilities that the medium presents with conventions that it enforces.  Writing is a powerfully diverse art.  A lot of it's good, a lot of it's bad.  "Blogging", however, is more specific, and unfortunately implies a sort of perpetual half-finished conversation.  It calls to mind a semi-private, informal, ephemeral, link-heavy style of extremely short-form writing.  This form has its masters: Tycho of Penny Arcade infamy leaps to mind immediately.  It also has a sea of mediocrity.  Statistically speaking, you can probably click the 'next blog' link at the top of this page for an immediate example.  I don't have a problem with any of this.  Even the "mediocrity" is just evidence of the degree to which this is empowering people: much of what I'd consider "mediocre" just isn't relevant to me, and isn't written for me.

But blogs can be, and are, so much more than that.  They are a disruptive technology in the world of publishing, where any style of writing can easily be published, circulated, and promoted.  One can write an entire novel, serialized chapter-by-chapter as blog posts.  Many people have, in fact, done this already.  You don't even need to emulate older forms of writing to step outside the style implied by "blogging".  The tools that the web affords — instant publishing, hyperlinks — are ideal for collaborative scientific research.  Hyperlinks take the work out of footnoting.

Prominent web writers who I respect also seem to avoid the use of the term "blog".  Joel Spolsky refers to other people's blogs, but the term "blog" does not appear anywhere describing his site, despite the fact that there is quite a bit of self-descriptive text that refers to "this site".  Paul Graham goes a step further, foregoing many traditional blog trappings and has a link that says, simply, "Essays".  I wonder if it's for this reason.

So, if you need to explain to someone who doesn't quite get what all the whole "blogging" thing is about, don't talk about social dynamics and the singularity and the mass popularization of media.  That's all great stuff, but it's a confusing distraction.  It's just like writing a book — or, more likely, a magazine.  Except you don't have to talk to a publisher.  And you don't have to have an editor.  And it's free.  And the publishing part doesn't actually take any time.  And it's accessible from anywhere in the world.  And you can read it on your cell phone.  When you stack up all the advantages, the lack of some bound paper doesn't seem like a big deal.

If you find this explanation useful, feel free to point your relatives at this post.  Tell them that you saw it on my blog, but don't tell them I blogged about it.  Tell them I wrote about it.

Conference FAIL

Last night at a dinner with Ivan Krstić and Itamar Shtull-Trauring, we were all lamenting that too many (all?) software conferences focus specifically on positive results. This is what you want, of course, if you treat a conference as purely a marketing venue. However, most learning takes place based on something that someone did wrong and then needed to correct, not something that they did right.

All of the great software developers I know have at least one great story of how a project they were working on was a complete disaster.  Often these projects are shielded from the public eye, since nobody wants to talk about failure.  So, how do we make a public discussion of these ideas socially acceptable?

Thus, an idea was born: FAILcon.  The idea is simple: submitted talks and papers must be related to projects which failed in an interesting way.  The larger the better, of course — the bigger they are, the harder they fail — but anything that failed in an interesting way would be a valid subject for discussion.

I'm writing about it so that it won't be forgotten, because I think it's a great idea.  But I doubt that any of us are going to organize a conference any time soon.  So please, steal this idea.  Does anyone out there with conference-organizing skills want to get something together based around the common theme of failure?