One of the things that confuses me most about Twisted is the fact that so many people seem to be confused by things about Twisted.
Much has been written, some of it by me, some of it by other brilliant members of the community, attempting to explain Twisted in lots of detail so that you can use it and understand it and control it to do your bidding. But today, I'd like to try something different, and instead of trying to help you figure out how to use Twisted, I will try to help you understand what Twisted is. To aid you in meditating upon its essence and to understand how it is a metaphor for software, and, if you are truly enlightened, for all life.
Let us contemplate the Twisted Way.
|
| Image Credit: Ian Sane |
In the beginning was the Tao.
All things issue from it; all things return to it.
-
Tao Te Ching
All information systems are metaphors for the world. All programs are, at least in small part, systems. Therefore, every program contains within it a world of thought. Those thoughts must be developed before they can be shared; therefore, one must have an interesting program before one thinks to do any interesting I/O.
It is less fashionable these days to speak of "object-oriented modeling" than it once was, but mostly because object-oriented design is now so pervasive that no-one needs convincing any more. Nevertheless, that is what almost all of us do. When an archetypical programmer in this new millennium sets out to create a program, they will typically begin by creating a Class, and then endowing that Class with Behavior; then, by creating an Instance of that Class, and bestowing interesting Data upon that Instance.
Such an Instance receives (as input) method calls from some other object, and produces (as output) method calls on some other object. It is a system unto itself, simulating some aspect of human endeavor, computing useful results and dispensing them, all in an abstract vacuum.
But, the programmers who produced this artifact desires it to interact with the world; to produce an effect, and therefore to accept Input and dispense Output.
It is at this point that the programmer encounters Twisted.
When you look for it, there is nothing to see.
When you listen for it, there is nothing to hear.
When you use it, it is inexhaustible.
Except that, in fact, nobody ever encounters Twisted this way. If this is where – and how – you encounter Twisted, then you will likely have great success with it. But everyone tends to encounter Twisted, like one encounters almost every other piece of infrastructure, in medias res. Method calls are flying around all over the place in some huge inscrutable system and you just have to bang through the tutorial to figure it all out right now, and it looks super weird.
Over the years, so many questions I've answered about Twisted seem to reduce to: "how do I even get this thing to do anything"?
This is Twisted's great mystery: it does nothing. By itself, it is the world's largest NOP. Its job, purely and simply is to connect your object to the world. You tell Twisted: listen for connections on this port; when one is made, do this. Make this request, and when it has a response, do that. Listen for email over SMTP; when one arrives, do the other.
Without your direction, reactor.run will just ... wait.
The source of most confusion with Twisted, I believe, is that few objects
are designed in this idiom. When we seek to create a program, we
feel we must start interacting with it immediately, before it even knows
what it is supposed to do. The seductions of blocking I/O are many
and varied. Any function which appears to merely compute a result
can simply be changed to cheat and get its answer by asking some other
system instead, with its callers none the wiser. Even for those of
us who know better, these little cheats accumulate and make the program
brittle and slow, and force it to be spun out into a thread, or the cold,
sparse desert of its own separate process, so it may tediously plod along,
waiting for the response to its each and every query.
Thus, desire (for immediate I/O) leads to suffering (of the maintenance
programmer).
Return is the movement of the Tao.
Yielding is the way of the Tao.
It doesn't need to be that way, though. When you create an object, it is best to create it as independently as possible; to test it in isolation; to discretely separate its every interaction with the outside world so that they may be carefully controlled, monitored, intercepted and inspected, one iteration at a time.
All your object needs to do is to define its units of work as atomic, individual functions that it wishes to perform; then, return to its caller and allow it to proceed.
The Master does their job and then stops.
They understand that the
universe is forever out of control.
When you design your objects by contemplating their purpose and making them have a consistent, nicely separated internal model of what they're supposed to represent, Twisted seems less like a straightjacket, contorting your program into some awkward shape. Instead, it becomes a comfortable jacket that your object might slip on to protect itself from the vicissitudes of whatever events may assault it from the network, whether they be the soft staccato of DNS, the confused warbling of SIP or the terrifying roar of IMAP.
Better yet, your object can be a model of some problem domain, and will therefore have a dedicated partner; a different object, a wrapper, whose entire purpose is to translate from a lexicon of network-based events, timers, and Deferred callbacks, into a language that is more directly applicable to your problem domain. After all, each request or response from the network means something to your application, otherwise it would not have been made; the process of explicitly enumerating all those meanings and recording and documenting them in a module dedicated to that purpose is a very useful exercise.
When your object has such unity of purpose and clarity of function, then Twisted can help manage its stream of events even if the events are not actually coming from a network; abstractions like deferreds, producers, consumers, cooperators and inline callbacks can be used to manipulate timers, keystrokes, and button clicks just as easily as network traffic.
True words aren't eloquent; eloquent words aren't true.
Sages don't need to prove their point;
those who need to prove their point aren't wise.
So, if you are setting out to learn to use Twisted, approach it in this
manner: it is not something that will, itself, give your object's inner
thoughts structure, purpose and meaning. It is merely a wrapper; an
interstitial layer between your logic and some other system. The
methods it calls upon you might be coming from anywhere. And indeed,
they should be coming from at least
one other place: your unit tests.
(With apologies to Geoffrey James and Laozi.)