A friend of a friend has had some trouble with the thugs who run the recording industry.

Whatever you feel about file-sharing, I hope we can all agree that it's wrong to legally attack, and bankrupt, a poor college student on a full scholarship, for copying four songs.

While I would, of course, love you all to donate to Twisted, if you only have fifty dollars to donate this year, give it to Fabiola.  We'll muddle through.

As bloggers are often wont to observe, the best thing about PyCon is the "hallway track".  This is especially true for old hands like me who are already familiar with most of the projects and techniques being introduced and described in the scheduled talks.

Usually I just wander around in the hallways and let random encounters occur as they will.  That's worked out reasonably well in the past, but I think I can get more out of the conference with a little preparation.  In particular, I'd like to use the open-space rooms to do some of this discussion so that we can really come to some conclusions rather than standing awkwardly in hallways, seeming as if we're about to leave.  So, with your help, I'd like to plan things out a little more.  Here are the things that I'd really like to do.

I am going to dedicate the sprint to thinking and talking about development on Twisted itself.  Of course I'm up for having a few meals during the conference with the Twisted developers who I don't get to see in person too often, but I talk to you guys all the time — I'd rather spend the conference talking to people I wouldn't otherwise see.

During the conference, I'd like to try to talk to people who are working on other projects.  Of particular interest to me are PyGame and Django.  There are currently no good, well-documented answers to "how do you use Twisted to make a networked game with PyGame" or "how do you use Twisted to network non-HTTP protocols with an existing Django app".  Both of those seem to be increasingly common questions.  If you are a leader of either of those projects, or can help me schmooze with one of them, I'd appreciate it.  I am, of course, just as interested in hearing from other projects that I don't even know about their twisted-integration stories.

In general I'd like to encourage other projects unrelated to Twisted to do this kind of cross-project jam session if you can.  There's a great opportunity for cross-pollenation at PyCon.

I'd also like to hear from users of Twisted, Nevow, Mantissa, et cetera, especially those who have not been particularly vocal on the mailing lists or IRC.  The best kind of user is a prospective volunteer, of course (stay for the sprints!), but both praise and constructive criticism are also welcome.

Don't underestimate the value of cheerleading!  Open source projects thrive on donated labor, and the fuel for donated labor is enthusiasm.  I can say for myself that a few kind comments at prior PyCons have motivated lots of hacking.  I don't just mean for me, though: take the opportunity to thank your favorite open source Python hacker, whoever that may be.

I'll be raising funds for the Twisted project.  If you're considering sponsorship, I would like to talk to you about how it will make you and everyone in your organization healthier, smarter, and better-looking.  (Mind-control powers and teleportation are reserved for sponsors at the gold level and higher, and only while supplies last.)

Finally, I'd like to talk about some of Divmod's less well-known projects, like Imaginary and Vertex, especially with people who are interested in helping to work on such things.  I am thinking about maybe doing a lightning talk or open space session, if it seems like there is enough interest to justify the preparation effort.

If any of these things are interesting to you, please leave a comment or drop me a note.  I doubt that I'm popular enough, or organized enough, to really do this properly, but I'd like to have at least a vague calendar of pre-planned meetings and open space sessions on my phone so I can avoid wasting any time once I'm actually there.

The only thing I take more seriously than my keyboards is my coffee.  I've been tinkering with my recipe for a while now, but I'm really very happy with the current iteration.  Here's exactly how I make my morning coffee.

1. Grind up roughly ¾ cup of 49th Parallel Epic Espresso using a Krups Fast Touch Coffee Grinder (Black).
2. Filter some water using a Brita Riviera water filter.
   
3. Place the grounds into an IKEA KAFFE french press, and fill it with filtered water.  Depress the plunger to make sure the grounds are soaked with water, then remove the plunger and cover with tinfoil.
4. Put the water into the refridgerator for between 12 and 24 hours.  (More time produces stronger coffee, but after 24 hours it starts to get unpleasantly bitter, and after 72 it's undrinkable.)
   
5. Put the plunger back in, and filter the espresso concentrate.  Decant it into another container.  (This concentrate will last for a week or two.)
6. To serve, mix 1 part concentrate with 3 parts Stonyfield Farm Organic Milk and 1 teaspoon of Butternut Mountain Farm Pure Vermont Maple Syrup.

Disclosure: nobody actually paid me to mention all those products.  I just like hyperlinks.

Jonathan Lange suggests that you have three options when you have some code that needs testing.

1. Give up
2. Work hard to write the damn test
3. Make your code testable.

In fact he starts with just the first two options and then reveals the third, but I suggest that there is an option number four, available only in Python (and other, similar dynamic languages):

4. Cheat.

I agree with Jonathan that you should generally make your code more flexible, and thereby testable.  Any code which is friendly to being invoked as a library or as an independent unit of functionality can be tested, so you should endeavor to never write code that is unfriendly to being executed as just a normal method call.  This is especially true if you are writing new code in a properly test-driven manner.  Keeping in mind, of course, that one should not modify the SUT purely for the purpose of testing.

However, if one is (as I often find myself) adding test coverage to a grotty old system, written at a time or in a place where test-driven development was not the norm, one typically wants to establish test coverage before making any changes to the code or its design.  In such a situation, one may often find oneself in the undesirable position of needing to carefully modify some implementation code so that it can be tested, hoping that none of its untested interactions with other areas of the system will be broken as a result.  For example, you might encounter some paranoid and misguided Java code like this:

    // Startup.java
    private static final void emitLogMessage(final String message) {
        System.out.println(message);
    }

    public static final void startUp() {
        // ...
        emitLogMessage("Starting up!");
        // ...
    }

In this case, it's very difficult to get in the way of any part of this system.  Nothing is parameterized, everything is global, and the compiler won't even let you call one of these methods.  You really only have Jonathan's three options here, none of which are desirable.

1. You can give up on testing this part of the system until you've covered other parts of the system.  In many cases this is the right thing to do, but it is often the lowest-level and most critical parts of a system which have calcified into this sort of untestable rubble.
2. You can work hard to write the damn test.  There are a number of extremely subtle nuances of the Java runtime which you can take advantage of to make a lie of the "private" and "final" keywords.  You can load the code using a custom classloader, manipulate its bytecode, or invoke private methods using reflection.  This is ultimately the "right thing" to do, but it requires the development of a daunting skill-set which you would not otherwise need.
3. You can make the code testable, changing it before you've properly tested the code which is already in use.  Ultimately this is what you want to get to anyway, but if the code is doing something subtle that you didn't test (and none of the rest of the system is tested yet) you might be (rightly) concerned that this could break something else.

In Python, the hard work is not so hard.  To start with, Python doesn't have the misfeatures of private and final.  It also doesn't have any baroque "reflection" constructs.  All you need to understand is attribute access.  So, if you have a similar Python file:

# startup.py
import sys

def emitLogMessage(message):
    sys.stdout.write("%s\n" % (message,))

def startUp():
    # ...
    emitLogMessage("Starting up!")
    # ...

Idiomatically, the situation looks just as hopeless.  Everything is global, and nothing is parameterized.  It's hard-coded.  However, if you look at it from the right angle, you will realize that you can't really code that "hard" in python.

What emitLogMessage is doing in this case is not making a fixed reference to the global sys module: it is simply accessing the sys attribute of the startup module.  So in fact, we can easily test it:

# test_startup.py
import sys
import startup
import unittest

class FakeSys(object):
    def __init__(self, test):
        self.test = test
    @property
    def stdout(self):
        return self
    def write(self, message):
        self.test.messages.append(message)

class StartupTest(unittest.TestCase):
    def setUp(self):
        self.messages = []
        startup.sys = FakeSys(self)

    def tearDown(self):
        startup.sys = sys

    def test_startupLogMessage(self):
        startup.startUp()
        self.assertEquals(self.messages, ["Starting up!\n"])

So testing startUp is a simple matter of replacing the sys object that it's talking to: which, in Python, is rarely more than a setattr() away.

I've taken care here to use only standard Python features.  This is, after all, theoretically possible in Java, it's just a heck of a lot harder, both to use and to understand — the learning curve is a big part of the problem.  However, if you're using Twisted, and willing to spend just a brief moment to learn about one of its testing features, you can save a few lines of code and opportunities for error:

import startup
from twisted.trial import unittest

class FakeSys(object):
    def __init__(self, test):
        self.test = test
    @property
    def stdout(self):
        return self
    def write(self, message):
        self.test.messages.append(message)

class StartupTest(unittest.TestCase):
    def setUp(self):
        self.messages = []
        self.patch(startup, "sys", FakeSys(self))

    def test_startupLogMessage(self):
        startup.startUp()
        self.assertEquals(self.messages, ["Starting up!\n"])

Please keep in mind that this is still not the best way to do things.  Use the front door first.  It's much better to use a stable, documented, supported API in your tests than to depend on an accident of implementation which should be able to change.  However, it is even worse to associate the feeling of testing with the feeling of being stuck, being unable to figure out how to dig yourself out of some hole that old, bad design has dug you into.

I'm writing this mostly for people who are new to test-driven development in Python and think that unit tests need to be a huge amount of extra work.  They don't.  If you ever find yourself struggling, unable to figure out how you could possibly write a test which would exercise some tangle of poorly-designed code, just remember: it's all just objects and methods, attributes and values.  You can replace anything with anything else with a pretty trivial amount of effort.  Of course you should try to figure out how to improve your design, but you should never think that you need to stop writing tests just because you used a global variable and you can't figure out what to replace it with.

Why?

Why use interfaces? Especially with Python's new ABCs, is there really a use for them?

Some of us Zope Interface fans — names withheld to protect the guilty, although you may feel free to unmask yourselves in the comments if you like — have expressed frustration that ABCs beat out interfaces for inclusion of the standard library.  However, I recently explored various mailing lists, Interfaces literature, and blogs, and haven't found a coherent description of why one would prefer interfaces over ABCs.  It's no surprise that Zope's interface package is poorly understood, given that nobody has even explained it!  In fact, PEP 3119 specifically says:

For now, I'll leave it to proponents of Interfaces to explain why Interfaces are better.

It seems that nobody has taken up the challenge.

I remember Jim Fulton trying to explain this to me many years ago, at a PyCon in Washington DC.  I definitely didn't understand it then.  I was reluctant to replace the crappy little interfaces system in Twisted at the time with something big and complicated-looking.   Luckily the other Twisted committers prevailed upon me, and Zope Interface has saved us from maintaining that misdesigned and semi-functional mess.

During that explanation, I remember that Jim kept saying that interfaces provided a model to "reason about intent".  At the time I didn't understand why you'd want to reason about intent in code.  Wouldn't the docstrings and the implementation specify the intent clearly enough?  Now, I can see exactly what he's talking about and I use the features he was referring to all the time.  I don't know how I'd write large Python programs without them.

Caveat

This isn't a rant against ABCs.  I think ABCs are mostly pretty good, certainly an improvement over what was (or rather, wasn't) there before.  ABCs provide things that Interfaces don't, like the new @abstractmethod and @abstractproperty decorators. Plus, one of the irritating things about using zope.interface is that the metadata about standard objects in zope.interface.common is not hooked up to anything: IMapping.providedBy({}) returns False.  ABCs will provide that metadata in the standard library, making zope.interface that much more useful once it has been upgraded to understand the declarations that the collections and numbers modules provide.

So, on to the main event: what do Zope Interfaces provide which makes them so great?

Clarity

Let's say we have an idea of something called a "vehicle".  We can represent it as one of two things: a real base class (Vehicle), an ABC (AVehicle) or an Interface (IVehicle).

There are a set of operations that interfaces and base-classes share.  We can ask, "is this thing I have a vehicle"?  In the base-class case we spell that if isinstance(something, Vehicle).  In the interfaces case, we say if IVehicle.providedBy(something).  We can ask, "will instances of this type be a vehicle?".  For an interface, we say if IVehicle.implementedBy(Something), and for a base class we say issubclass(Something, Vehicle).  With the new hooks provided by the ABCs in 2.6 and 3.0, these are almost equivalent.  With zope.interface, you can subclass InterfaceClass and write your own providedBy method.  With the ABC system, you subclass type and implement __instancecheck__.

However, there are some questions you can't quite cleanly ask of the ABC system.  For one thing, what does it really mean to be a Vehicle?  If you are looking at AVehicle, you can't tell the difference between implementation details and the specification of the interface.  You can use dir() and ignore a few of the usual suspects — __doc__, __module__, __name__, _abc_negative_cache_version — but what about the quirkier bits?  Metaclasses, inherited attributes, and so on?  There's probably some way to do it, but I certainly can't figure it out quickly enough to include in this article.  In other words, types have two jobs: they might be ABCs, or they might be types, or they might be both, and it's impossible to separate those responsibilities.

With an Interface, this question is a lot easier to ask.  For a quick look, list(IVehicle) will give a complete list of all the attributes expected of a vehicle, as strings.  If you want more detail, IVehicle.namesAndDescriptions() and Method.getSignatureInfo() will oblige.

Since the interface encapsulates only what an object is supposed to be, and no functionality of its own, it's possible for frameworks to inspect them and provide much nicer error messages when objects don't match their expectations.  zope.interface.verifyClass and zope.interface.verifyObject can tell you, both for error-reporting and unit-testing purposes, whether an object looks like a proper vehicle or not, without actually trying to drive it around.

Flexibility

At the most basic level, interfaces are more flexible because they are objects.  ABCs aren't objects, at least in the message-passing smalltalk sense; they are a collection of top-level functions and some rules about how those functions apply to types.  If you want to change the answer to isinstance(), you need to register a type by using ABCMeta.register or overriding __instancecheck__ on a real subclass of type.  If you want to change the answer to providedBy, for example for a unit test, all you need is an object with a providedBy method.

Of course, you can do it "for real" with an InterfaceClass, but you don't need to.  In other words, its semantics are those of a normal method call.

Interfaces aren't completely self-contained, of course: there are top-level functions that operate on interfaces, like verifyObject.  However, there's an interface to describe what is expected:

>>> from zope.interface.interfaces import Interface, IInterface
>>> IInterface.providedBy(Interface)
True
There's also the issue of who implements what.  For example, you might have a plug-in system which requires modules to implement some functionality.  Generally speaking, modules are instances of ModuleType, so specifying that all modules implement some type with an ABC is somewhat awkward.  With an interface, however, there is a specific facility for this: you put a moduleProvides(IVehicle) declaration at the top of your module.

In zope.interface, there is a very clear conceptual break between implements and provides.  A module may provide an interface — i.e. be an object which satisfies that interface at runtime — without there being any object that implements that interface — i.e. is a type whose instances automatically provide it.  This distinction comes in handy when avoiding certain things.  This distinction exists with ABCs; either you "are a subclass of" a type or you "are an instance of" a type, but the language around it is more awkward and vague, especially since you can be a "virtual instance" or "virtual subclass" now as well.

There's also the issue of dynamic proxies.  If you have a wrapper which provides security around another object, or transparent remote access to another object, or records method calls (and so on) the wrapper really wants to say that it provides the interfaces provided by the object it is wrapping, but the wrapper type does not implement those interfaces.  In other words, different instances of your wrapper class will actually provide different interfaces.  With zope.interface you can declare this via the directlyProvides declaration.  With ABCs, this is not generally possible because ABCMeta.register will only work on a type.

Adaptation

Let's say I have an object that provides IVehicle.  I want to display it somehow — and in today's web-centric world, that probably means "I want to generate some HTML".  How do I get from here to there?  ABCs don't provide an answer to that question.  Interfaces don't do that directly either, but they do provide a mechanism which allows you to provide an answer: you can adapt from one interface to another.

I'm not going to get into the intricacies of exactly how adaptation works in zope.interface, since it isn't important to understand most of the time.  Suffice it to say you can adapt based on specific hooks that are registered, based on the type an object is, or based on what interfaces it provides.

The gist of it is that you have some thing that you don't know what it is, and you want an object that provides IHTMLRenderer.  The way you express that intent is:

renderer = IHTMLRenderer(someObject)

If there are no rules for adapting an object like the one you have passed to an IHTMLRenderer, then you will get an exception - which is all that will happen, normally.  However, this point of separation between the contract that your code expects and the concrete type that your code ends up actually talking to can be very useful.

The larger Zope application server has a rich and complex set of tools for defining which adapter is appropriate in which context, but Twisted has a very simple interface to adaptation.  You simply register an adapter, which is a 1-argument callable that takes an object that conforms to some interface or is an instance of some class, and returns an object that provides another interface.  Here's how you do it:

import attr
from zope.interface import implementer
from twisted.python.components import registerAdapter

@implementer(IHTMLRenderer)
@attr.s
class VehicleRenderer(object):
    "Render a vehicle as HTML"
    vehicle = attr.ib(validator=attr.validator.provides(IVehicle))
    def renderHTML(self):
        return "<h1>A Great Vehicle %s (%s)</h1>" % (
                   self.vehicle.make.name,
                   self.vehicle.model.name)

registerAdapter(VehicleRenderer, IVehicle, IHTMLRenderer)

Now, whenever you do IHTMLRenderer(someVehicle), you'll get a VehicleRenderer(someVehicle).

Your code for rendering now doesn't need any special-case knowledge about particular types.  It is written to an interface, and it's very easy to figure out which one; it says "IHTMLRenderer" right there.  It's also easy to find implementors of that interface; just grep for "implementer.*IHTMLRenderer" or similar.  Or, use pydoctor and look at the "known implementations" section for the interface in question.

Conclusion

In a super-dynamic language like Python, you don't need a system for explicit abstract interfaces.  No compiler is going to shoot you for calling a 'foo' method.  But, formal interface definitions serve many purposes.  They can function as documentation, as a touchstone for code which wants to clearly report programming errors ("warning:  MyWidget claims to implement IWidget, but doesn't implement a 'doWidgetStuff' method"), and a mechanism for indirection when you know what contract your code wants but you don't know what implementation will necessarily satisfy it (adaptation).

Even with a standard library mechanism for doing some of these things, Zope Interface remains a highly useful library, and if you are working on a large Python system you should consider augmenting its organization and documentation with this excellent tool.