It occurs to me that there may still be a few Python people who read this blog but have not yet discovered JP Calderone's.

If you are such a person, he just did an excellent write-up of the practical implications of Python's rich comparison operators.  Check it out.

Have you ever written a module that looked like this?

subscribers = []

def addSubscriber(subscriber):
    subscribers.append(subscriber)

def publish(message):
    for subscriber in subscribers:
        subscriber.notify(message)


And then used it like this?

from publisher import publish

class worker:
    def work(self):
        publish(self)


I've done this many times myself.

I used to think that this was the "right" way to implement Singletons in Python.  Other languages had static members and synchronized static accessors and factory methods; all kinds of rigamarole to achieve this effect, but Python simply had modules.

Now, however, I realize that there is no "right" way to implement Singleton in Python, because singletons are simply a bad thing to have.  As Wikipedia points out, "It is also considered an anti-pattern since it is often used as a euphemism for global variable."

The module above is brittle, and as a result, unpleasant to test and extend.

It's difficult to test because the call to "publish" cannot be indirected without monkeying around with the module's globals - generally recognized to be poor style, and prone to errors which will corrupt later, unrelated tests.

It makes code that interacts with it difficult to test, because while you can temporary mangle global variables in the most egregious of whitebox tests, tests for code that is further away shouldn't need to know about the implementation detail of "publish".  Furthermore, code which adds subscribers to the global list will destructively change the behavior of later tests (or later code, if you try to invoke your tests in a running environment, since we all know running environments are where the interesting bugs occur).

It's difficult to extend because there is no explicit integration point with 'publish', and all instances share the same look-up.  If you want to override the behavior of "work" and send it to a different publisher, you can't call to the superclass's implementation.

Unfortunately, this probably doesn't seem particularly bad, because bad examples abound.  It's just the status quo.  Twisted's twisted.python.log module is used everywhere like this.  The standard library's sys.path, sys.stdin/out/err, warnings.warn_explicit, and probably a dozen examples I can't think of off the top of my head, all work like this.

And there's a good reason that this keeps happening.  Sometimes, you feel as though your program really does need a "global" registry for some reason; you find yourself wanting access to the same central object in a variety of different places.  It seems convenient to have it available, and it basically works.

Here's a technique for implementing that convenience, while still allowing for a clean point of integration with other code.

First, make your "global" thing be a class.

class Publisher:
    def __init__(self):
        self.subscribers = []

    def addSubscriber(self, subscriber):
        self.subscribers.append(subscriber)

    def publish(self, message):
        for subscriber in self.subscribers:
            subscriber.notify(message)

thePublisher = Publisher()

Second, decide and document how "global" you mean.  Is it global to your process?  Global to a particular group of objects?  Global to a certain kind of class?  Document that, and make sure it is clear who should use the singleton you've created.  At some point in the future, someone will almost certainly come along with a surprising requirement which makes them want a different, or wrapped version of your global thing,  Documentation is always important, but it is particularly important when dealing with globals, because there's really no such thing as completely global, and it is difficult to determine from context just how global you intend for something to be.

Third, and finally, encourage using your singleton by using it as a default, rather than accessing it directly.  For example:

from publisher import thePublisher

class Worker:
    publisher = thePublisher

    def work(self):
        self.publisher.publish(self)


In this example, you now have a clean point of integration for testing and extending this code.  You can make a single Worker instance, and change its "publisher" attribute before calling "work".  Of course, if you're willing to burn a whole extra two lines of code, you can make it an optional argument to the constructor of Worker.  If you decide that in fact, your publisher isn't global at all, but system-specific, this vastly decreases the amount of code you have to change.

Does this mean you should make everything into objects, and never use free functions?  No.  Free functions are fine, but functions in Python are for functional programming.  The hint is right there in the name.  If you are performing computations which return values, and calling other functions which do the same thing, it makes perfect sense to use free functions and not bog yourself down with useless object allocations and 'self' arguments.

Once you've started adding mutable state into the mix, you're into object territory.  If you're appending to a global list, if you're setting a global "state" variable, even if you're writing to a global file, it's time to make a class and give it some methods.

Often it is the means that justify the ends: Goals advance technique and technique survives even when goal structures crumble.

- Alan Perlis, Epigram 64, "Epigrams in Programming"

How does a computer programmer get better at computer programming?

In a very broad sense, there is some consensus on the answer to this question.  It's like the answer any other skill.  "Do a lot of it."  Much like other skills, though, while necessary, this condition is not sufficient.  If you want to get really good at punching, you can punch a brick with your thumb inside your first ten thousand times, and someone who has learned to do it correctly and had a fraction of the equivalent amount of practice will be a lot better at punching than you.

So, to expand on that consensus, to get better at programming, you have to learn to do it right, and then do a lot of it, hopefully with someone correcting your mistakes and helping you do it better each time.  Like any other kind of serious training.

This concept is pretty old.  Shortly after the dawn of time, the dragon god-emperor pioneered this concept when he invented a martial art that is now popularly known as "Kung Fu", or, literally, "achievement through great effort".  The term does not necessarily refer specifically to martial arts, and can (in both Chinese and English) refer simply to the fact that someone has achieved something exceptional through a lot of effort and practice.

Specious metaphors between computer programming and martial arts (and various¹ eastern philosophies) have existed for almost as long as computer programming itself.  As a cocky and callow youth, I occasionally had these metaphors generously applied to me, but they rang hollow even in my hubristic state of mind.  I knew that despite any talents I might have, at the very least, I hadn't spent a lifetime of effort achieving a near-supernatural mastery of computers.  I had just messed around with C++ and Java a little bit.  The appellation of "master" was inappropriate, but I didn't know how one really did go about getting to that point.  I had already met a few folks in the programming profession who were old, but certainly not yet wise, and I wanted to learn how not to end up like that.

Whenever two programmers meet to criticize their programs, both are silent.

- Alan Perlis, Epigram 101, "Epigrams in Programming"

I first noticed that the parallel between UQDS and these various arts and philosophies when a friend observing Divmod from afar mentioned that our review process was intimidating.  Intrigued, I conducted a straw poll among my programming friends who are not deeply immersed in the eternal struggle of the Twisted or Divmod codebases, and found that there was a fairly consistent impression: the exchange of comments between authors and reviewers was highly unusual, intimidating, and impressive.

It's unusual because most people cannot read code well enough to comment in as much depth.  Code review is becoming more popular, but it's still rare.  Even intense, newfangled "Agile" processes like Extreme Programming focus more on discussing code as it's being created than reading code which has already been written.

The exchange is intimidating because many of the comments are extremely fine-grained and not very forgiving.  It isn't uncommon to see comments about a tiny private method missing documentation, or test coverage doesn't hit a single error condition.  This causes the reader to muse on the ways in which their own code would fail if it were subjected to such intense scrutiny.

Finally, the exchange is impressive because the factors which make it intimidating make the programmers doing it seem extremely skilled.  If you're looking at work which is being subjected to criticism only on details which seem insignificant, it implies that it is in every other way correct.  To some extent this impression is illusory, because the reviewers are only human and may be missing more important things.  For something as complex and intricate as software, though, it's impressive assuming that the reviewers tried at all.

I can't pretend that UQDS is an entirely unique process in this regard.  The Python and Subversion projects practice regular code reviews in public, and those are just the ones I know off the top of my head.  This isn't a phenomenon unique to open source either - I know that both Canonical and Google do fairly comprehensive code review on their closed-source products.  I've heard milder expressions of awe - although perhaps they were only milder because they came from friends - about those projects and companies.

While it isn't unique, UQDS is particularly intense.  Some other processes try to be "reasonable" - which is, on the face of it, a reasonable thing to do.  Small, "obviously correct" fixes are sometimes allowed without tests.  "Security" fixes are sometimes allowed in for expediency because of their urgency.  Some committers are sometimes exempt.  Some reviews are done after the fact.  Documentation is often optional, since reviews are meant to enforce correctness, not every aspect of quality.  For better or worse, UQDS sets out a much stricter guidelines: everything must be tested.  Everything must be documented.  Nobody is allowed to bypass the process.

It isn't all roses.  Especially when a contributor needs to update an area of code not originally developed under this regime, it can be frustrating to need to document and test and correct code that isn't really related to their goal.  Sometimes the overabundance of the opportunity for review creates interminable discussions about a design, when there is really a need to just go with a solution that works for now.  There may be some room to adjust the constraints to place less of a burden on those who want to contribute small improvements, and that's difficult without compromising on overall quality.

Despite their drawbacks, these rules plus a public ticket tracker add up to something that the industry of martial arts has known about the power of for quite some time: the exhibition.  If you're going to run a martial arts studio you need to attract students, and to do that you need to "wow" people every so often.  Martial arts are about defense, mind-body integration, health and fitness, and self-discipline, but putting that on a poster isn't going to fill up the thursday evening class.  Sometimes you need to get your school together and show off just how bad you would be able to kick some ass with your skills, you know, just in case anybody was wondering.

I am quite proud of the fact that I've received numerous unsolicited comments about the quality of the code itself within certain parts of Divmod and Twisted.  Like any programmer I've received my share of complaints about my work, too (especially the stuff that precedes all this intense review), but as Steve Yegge famously observed, most programmers who are famous for their programs don't actually write programs; they write about programs, or they wrote some programs nobody has the code to.

I'm almost more pleased with the idea that I could help pave the way for programming to have some real rock-stars on the merit of their code than the idea of being a rock-star myself.

Dealing with failure is easy: Work hard to improve. Success is also easy to handle: You've solved the wrong problem. Work hard to improve.

- Alan Perlis, Epigram 101, "Epigrams in Programming"

Or, 72/72 to be precise.

Twisted is once again in danger of losing the engineship of your Internet!  If you haven't voted on this Jyte claim already, please do so now!

Unfortunately, if you have questions about how to write and run tests using Twisted's "trial" tool, this work in progress is still the best answer I can give you.  It's a good work in progress, but it's definitely not finished.

If you need some documentation, please feel free to provide feedback and patches at Twisted's ticket number 2443.

I am working on putting together some ramblings about test-driven development of my own, but at this rate it's going to be quite a while until I'm ready to post them here.