I am a fan of interactive fiction.  While it would be an understatement to say that the medium has experienced a bit of a lull since its heyday in the era of Infocom, I have been fairly impressed by the IFComp competitions of recent years; really enjoying these games as new, unique experiences rather than nostalgia or kitsch.

One thing has been consistently irritating is getting all of my fancy, modern hardware to play these games in a satisfying way.  The one place I definitely don't want any nostalgia in this experience is remembering the 14-inch flickery CRTs that I played these games on in my youth.  I could point fingers at various unsatisfactory pieces of software, but the fact is that the increasing variety of interpreters (glulxe, hugo, tads and z-machines of various versions) it is sometimes frustrating to even get these games to run on linux, let alone look good.

Enter Gargoyle, a classy looking, multi-platform, multi-interpreter interactive fiction system.

My original experience with Gargoyle, several months ago, was actually pretty bad.  It looked nice, but it took forever to figure out how to compile it.  I had to patch it and I couldn't figure out how to do it gracefully enough to compile on anybody else's system.  When I did built it, it opened a small, apparently fixed-size window with a fairly small font.  Not that good for an immersive experience on a desktop, and terrible for playing on the couch on my HDTV across the room.

Today, I discovered that, first of all, it builds now.  Not only that, there are some packages for Ubuntu if you are adventurous enough to try them.  Finally, and perhaps most importantly, I discovered that you can configure it pretty easily.  In a few minutes, thanks to the "Toggle Fullscreen" key in Compiz's "Extra WM Actions" plugin, I had a config file which looks really nice either on a "full HD" (1920x1080) TV or WUXGA desktop monitor.

Thanks to Gargoyle, here is what Zork looks like today:


Get your copy of "Zork HD" (by which I mean, "my ~/.garglkrc file") here.  Toggle it full-screen and enjoy.

I use Emacs.  However, unlike some Emacs users, I don't treat it as a religion.  In fact, I'd rather be using a more "modern" IDE; one that understands my code on a deeper level and provides things like refactoring tools, integrated debugging, and "view method implementation" that work reliably and don't require weeks of configuration effort to use.  One that uses modern UI conventions instead of arcana from the 70s so that my friends who are not emacs-heads can quickly wrap their heads around what's going on on my screen, and perhaps dare to touch my keyboard.

However, even if one is keen to do it, switching away from Emacs is a big deal.  I see lots of editors that advertise "emacs keybindings".  While I appreciate the effort, these features always look like someone who has no idea how to use Emacs worked through some kind of quick cheat-sheet of features like "keybinding for Save", "keybinding for Save As", "keybinding for Close Window" and just added them one after another.  Sometimes, with no regard for whether these keys conflict with other shortcuts!  (I'm looking at you, gnome "key-theme".)

Do you think you can write an editor which can replace Emacs for me?  Here are a few features, taken both from my years of customizing Emacs to meet my needs and some basic features in Emacs itself that non-natives never seem to understand.

I'm leaving out the extremely basic stuff like "syntax highlighting" and "automatic indentation" since most editors do OK on those fronts already.  These are the things that I find have been subtle, in that they're broken almost everywhere outside of emacs.

Can you do what I mean when I press the "go" button?

When I edit code, I repeat these steps endlessly:

1. Edit a test file.
2. Run the tests.
3. Edit an implementation file.
4. Run the tests.

In order to do this in my current emacs setup, I open my implementation file, then open my test file, then press F9.  Then, I edit a little bit, and press F9 again.  Then, I switch to the implementation file, type a little bit, and press F9 again - Emacs knows which tests to run, because the implementation file has an annotation in it which describes the test-cases that are associated with it.

When I'm done, I push F5 and it immediately jumps my cursor to the error that i'm working on.

This works for me in Java, in C, and in Python.  I've got a little bit of custom emacs-lisp code that I wrote to do each of them.  I'm willing to write a little more.  But, is it very easy in your editor to grab a global keybinding?  To write a plugin in 4 or 5 lines of code that just formats a command-line string to run?  To parse the output of a subprocess?  To visit a file and line number without requiring further user interaction?

Can I reach the thing I need to work on fast?

Let's say I'm working on a file called "foo.c".  I want to open "bar-baz-boz-qux.c" in the same directory.  In Emacs, this is probably just "control-x control-f b <tab> <enter>" - maybe a few more letters if there are other "bar-" files.  Do I need to hit more buttons than that in your editor?  Do I need to reach for the mouse?  Do I need to navigate the inefficient-even-from-the-keyboard GTK file dialog?

Now, let's say I've got the file "foo.c" open in several different branches of the same project, and I want to quickly alternate between them, display them side-by-side, etc.  In Emacs, "C-x C-b" will bring up a list of every open file, and as I type its name, the list is reduced to only those who match what I've typed so far.  So if, for example, I type "foo.c", I'll get all my "foo.c"s.  I can cycle between them with C-s until I get the one that I want.

I don't want to hunt around and click on a tab-bar or hit "next next next next".  I just want to type in a little bit of the file name and have the editor figure the rest out for itself.

Can I use it on Mac, Linux, FreeBSD, Solaris, and Windows?

Emacs is extremely portable.  Every operating system I want to work on (and many, many that I hope I never do) can run it.  I don't want to invest the energy to learn a new editor if I can't use it everywhere.

Can I use it remotely over the internet?  Collaboratively?

You can cheat on this one: if you can run in a terminal (and under Screen), then you get both of these for free.  Emacs cheats that way.  But if you can't run in a terminal - no, I'm sorry, VNC is not an acceptable substitute.  It doesn't perform adequately over the internet and it probably never will - as the internet gets faster, the average display gets bigger and has more colors on it.

Your editor probably can't run in a terminal.  So you'd better give me a way to pair-program with people over the internet and a way to access the editor on my desktop machine when I'm away from home.

Can I see whitespace?

I don't like to leave invisible droppings in files when I edit them.  I'd like to be able to see trailing whitespace, highlight it, and eliminate it.  I'd like to be able to tell if I have any tabs in my files (python does not like that very much).

Do I need to carefully juggle my clipboard - or do you have a "kill ring"?

Normally, if you cut some text, then cut some other text, you lose the first text — unless you use "undo" or something like that and screw up your editing state.

In Emacs, when I cut five or six different pieces of text, and I go to paste them, I can paste any of them.  I don't have to carefully remember what's on my clipboard, because the last 60 or so things that I cut or copied are around in case I need any of them.

Do I need to carefully remember where I was and scroll around to get back there - or do you have a "mark ring"?

Emacs has what amounts to a "back button" for your text editor.  If I edit something interesting, go to another window, go to another project, and then want to jump back a few steps, I can easily do that.

This is particularly helpful when, for example, inserting some import statements.  I'm in the middle of a function in the middle of a big file.  I want to use the Foo class, but it's not available yet.  So, I jump to the beginning of the buffer, type "from baz.bar.foo import Foo" and then hit "C-u C-space" to jump back to the middle of that function I was working on.  "C-x C-space" does something similar, but can even take me to different files.

Can you do smart word-wrapping?

/**
 * When I have a long documentation comment in ActionScript, JavaScript or
 * Java, Emacs will helpfully wrap it like this.  If I make changes in the
 * middle and then re-wrap it, it stays wrapped and helpfully adjusts the
 * placement of the asterisks along the left-hand side.  Can your editor do
 * this?
 */

# But when I have a comment in Python, it's formatted like this.  I don't have
# to tell Emacs anything about the different comment styles.

-- For that matter, it can understand and properly format SQL comments too.
-- And C/C++, and Ruby, and PHP, and more.

"""
If I format code inside a docstring, it flows properly too.  Granted, there are
a lot of bugs in this particular case in the stock Emacs, but since it works
everywhere else I have written some workarounds.  (You could always work around
it by inserting some extra blank lines before wrapping, but that always
bothered me.)  Can I customize how your flowing works if I don't like it?
"""

Is there version-control integration?

If I'm editing a project that uses bzr, darcs, git, svn, hg, perforce, or cvs, I can get a nice "status" page as a jumping-off point in Emacs to show me which files are in version control and what files I've edited.  I can update, commit, pull, push, and diff out without leaving the editor.  Can your editor do that?  And I don't just mean, can your editor do that for SVN.  Does it support all of the systems I just named and a few others for good measure?

Can I tell what I'm working on?

I don't like having to scroll around to figure out what function I'm in the middle of when I forget.  I work on a lot of code, and I browse a lot of code, and sometimes if I'm in the middle of a 200-line-long function I can't see the class or the function name.  Emacs has a feature called "which-func-mode" which allows me to glance at the bottom of the screen and instantly know what function I'm working on.  Fancy, glowy sidebars with tree-views of my whole source file and inheritance hierarchy are great, but can I always see the name of the class and the method that I'm working on now?  Even if there are so many other methods on that class that the fancy method list on the left has to scroll?

Last but not least...

Can I code for it?

I'm a programmer and I need a programmer's editor.  I don't want to write giant, heavyweight plugins; I want to be able to quickly toss off a snippet of code which modifies the editor.  But, I'm not an IDE developer.  I don't want to write a giant plugin; I want an editor which lets me organically grow my own modifications when I find myself doing some task frequently.

For example, I have my own "snippets"-type module, "quick-hack mode", which does a ton of clever-clever things like inserting

def (self):
    """
    """

when I type "def" inside of a "class" block.  (of course the "def" is omitted otherwise).

I have a hotkey to turn this off in case other people find it annoying ­— and it's difficult to be ambivalent about this mode, you either love it or hate it: it's a very personal thing, and I don't expect your editor to support it directly.  This mode was developed after years of observing my own peculiar tics while editing and crafting conveniences to support that and free me from distractions.

Since developing this kind of support code isn't my main interest, coding needs to be more than possible, it needs to be easy.  I need interactive help and the ability to load a brief snippet of code into the editor without restarting it; a reasonable debugger so when it blows up in my face I can at least sort of tell why.

Do you feel lucky?  Well, do you?

Emacs has a lot of features.  You don't have to replicate them all.  But if you want me and the millions like me — okay well maybe not millions, but it's not just me either — to switch to your shiny new editor, you'd better be doing all of these things and doing at least one or two other totally awesome things that emacs can't do.

Last month, a thread on the twisted-web mailing list got me thinking about a frequently implemented, but seldom understood usage of Twisted: writing applications backed by a traditional database server.  I tried to write a timely reply on the mailing list, but found what I had to say on the topic was overflowing the polite bounds of an email message.  I've tried to write about this before, buried in the middle of a post about something else.  I don't think I really got my message across, though, because I believe this was quoted as saying that I find asynchronous data-access APIs "extremely painful".  Asynchronousness is not the point that I find difficult, as much as I do transactionality (and integration with existing database bindings).

So this time, please bear with me as I explain enough context to properly frame my opinion.

I think that concurrency is a difficult problem that affects every aspect of your code, and so it is important to have a comprehensive, consistent, and easy to understand plan to deal with it in any given system.

Twisted's "cooperatively multitasking / callbacks-scheduled-by-I/O-and-timers" idiom is one concurrency model.  Deferreds are a super important convenience mechanism in that model, but they're not completely necessary; you can do this with just dataReceived, connectionLost, callLater etc.

In general this model - let's call it something memorable, like "CM/CSBIOAT" - is a pretty easy concurrency model to work with once you know how it works.  In particular, it's pretty easy to avoid making a common variety of serious concurrency mistakes, since you don't need to remember to declare any locks, and the behavior of the system under load is unsurprising, if not necessarily ideal for performance.

Threading is another concurrency model with which we are all familiar.  Shared-state multithreading is a pretty bad concurrency model for general use.  In particular, it's very easy to make mistakes that are impossible to diagnose or reproduce.  Despite its unsuitability for applications, threading can be a useful building-block as a low-level tool to construct higher-level concurrency models.  In many practical cases I am aware of, threading is the only available building-block at this level for building efficient implementations of other concurrency models, because operating systems and compilers don't provide anything better.

There is an antipattern that arises from a somewhat naive understanding of these two models.  The Twisted novitiate discovers that Twisted Is Good, and Threads Are Bad.  Experimentally they discover that this is indeed true, and that despite its eccentricities, writing and debugging "Twisted" code (whose benefits really come from the CM/CSBIOAT pattern) is a lot easier than writing and debugging threaded code.

So, our unfortunate Twisted novice now needs to write a database application: what to do?  Well, one way they can write it is to "just use threads" for data-access logic and communicate with Twisted some other how - for example, to put all their database logic in a function they pass to runInteraction.  The only other apparent option is to "use Deferreds" and invoke adbapi.ConnectionPool.runQuery or runOperation.  Deferreds are Twisted - Good!  Threads are ... threads!  Bad!  The answer seems obvious.

However, in choosing to use this facility, you've done far more than choosing between "twisted" and "threads".  If you use runInteraction, you can easily keep all of your work in a single transaction; since database APIs are blocking, you can only safely do a read followed by a write in the same transaction if you can block between those calls.  If you do a runQuery, take the result of that and pass it as input to a runOperation, you're sharing data between two different transactions and potentially two different cursors.  Whether Deferreds or good or not, this breaks the assumptions that the underlying database uses to keep its data consistent.  Consider incrementing a counter.  In the "threaded" case, you'd do something like this:

  def interaction(txn):
      x = doSql(txn, "select thingy from foo where bar = baz;")
      doSql(txn, "update foo set thingy = ? where bar = baz;", x+1)
  cp.runInteraction(interaction)

This always results in foo.thingy being set to foo.thingy + 1.  If your database is set up properly (and most are by default) there's no opportunity for other code to execute between those two statements.

But in the "twisted" case you do something like this:

  @inlineCallbacks
  def stuff(cp):
      x = yield cp.runQuery("select thingy from foo where bar = baz;");
      yield cp.runOperation("update foo set thingy = ? where bar = baz;", x+1)

As syntactically pleasant as that appears to be, and as convenient as it might seem to be able to call Twisted APIs as much as you want in the middle of this work, any amount of code can run between the first line and the second, thanks to those pesky 'yield's.  That means if you run 'stuff' twice, there's a good chance that your callbacks will stomp on each other and one of the increments will be lost.

Transactional relational database access is a really different concurrency model, all its own.  In many cases it appears to be the same as plain old shared-state multithreading; not least of which because it is implemented using threads and the threads are completely exposed to application code, and made part of the database interface's API.  However, using a transactional database to store your interesting state is much, much safer than just using threads to access any old datastructure.  An ACID database is specifically implemented to provide a consistent view of your data to any executing client at any time, and in the cases where that would be impossible, to schedule execution of various clients to provide an ordering where data is consistent.  (You'll notice that I have avoided saying "thread", but in practice an executing SQL client is a thread in your application.)  Caching middleware confuses this issue, making it more like regular multithreading; but in a good SQL database, using threads rather than just separate processes is just an optimization; one which should be completely transparent to your application code.

Axiom doesn't really have a concurrency model (it ought to, but that's a discussion for another day).  The idea there is that, like the rest of Twisted, you try hard never to block for too long.  It is possible — too easy, really — to screw this up and block for a long time waiting for the disk in an Axiom program, but to some extent that's true of any Python code.  Since Axiom is typically accessed by one, or at most two processes at a time, you won't end up blocking on your database for a long time because some other code is using it; the main thing Twisted's concurrency model is designed to prevent is your code blocking and getting stuck or being idle, not your code blocking at all.  So, I'm going to give you advice for using Storm or ADBAPI: the only advice for Axiom is "write fast queries".

Assuming that you're writing a traditional database application, here's my advice for you.

Let's assume that Storm (or ADBAPI) does not have any thread-safety bugs itself.  This assumption is unlikely to be completely true, but you probably have to make it regardless if you're going to use either of these things at all, regardless of my advice :).  With that assumption, you can use Storm (or ADBAPI) with Twisted from a thread-pool and pretend, in your application, that the threads do not exist.  You should avoid accessing global state and pretend that your code might be run in a subprocess or a thread or even on a different computer. If you're lucky, one day it will be, and your application will "magically" scale!  If you follow this simple discipline, you can cleanly interface between the Twisted concurrency model (where you do all of your non-database I/O) and the RDBMS concurrency model (where you interact with all of your "data" objects).

Don't touch any database objects in your Twisted mainloop.  Don't touch any Twisted objects in your database transactions.  This has the added benefit of not needing to worry that you're sending out information about partially-completed database operations to a network connection, or injecting potentially transient network state into a persistent database operation that may need to be re-tried.

In theory, there's nothing stopping an asynchronous data-access API from doing all of the same stuff that I just described threads doing.  All you'd need is good non-blocking database infrastructure, non-blocking transactions, and a bunch of code to associate a running transaction object with a particular database transaction and cursor.  It is possible, if you go down to the database-protocol level, to write a database wrapper which actually integrated with the Twisted concurrency model and treated your database as just another source of input and output.  In terms of preventing errors and assisting making code testable and deterministic, I think this would be an improvement over the threaded version of this solution.

However, implementing such an improvement would likely take quite a bit of time.  Time that most small database-backed projects don't have, so it's unlikely someone will need to scratch this particular itch any time soon.  Even if someone did do all that work for one database, it's likely that a lot of it would need to be done over again for each subsequent set of database bindings; so, using a DB-API module in a thread would remain the only way to retain database portability.

For the moment, threads and threadpools are the tools that existing database bindings give us to manage transactions, and it's likely that they're adequate for a huge majority of applications.  The only real problem is that you can't completely hide threads from the application and make sure they're not being used for evil.

I play video games.  Also, I'm married.  Ying also plays video games.  More than I do, even.  Where are the games — besides WoW — that we can play together?

I know a couple of other guys who like to play games with their significant others.  I really feel like the gaming generation has grown up at this point, but where are the grown-up games?

My favorite kind of video games are immersive, story-driven games with open worlds and a lot of flexibility.  I am really digging The Witcher: Fewer Bugs Edition right now, but despite its "mature" and "philosophical" themes, it feels like a game written for a "mature" and "philosophical" adolescent male misfit rather than the usual vanilla adolescent male misfit.  That's not really a black mark against this specific title - it in particular seems to pull off the stereotypical fantasy tropes very well.

While the independent gaming scene is a lot better in terms of raw originality, I haven't seen anything I can recall on TIGSource where I thought "That would be great to play with Ying."

This is mostly a rhetorical question (get to making those games, game-makers who are reading this!) but I would also be very happy to be proven wrong.  If you leave a comment with a game we end up enjoying I'll definitely blog about this again.