There's no such thing as magic. So when someone tells you that you can
magically transform blocking code into Deferreds, as in this Python
Cookbook posting, From
blocking functions to Deferred functions, you should be suspicious.
As Itamar suggested, this particular goal can be accomplished with Twisted's
standard
twisted.internet.threads.deferToThread, which lacks the horrible,
possibly crashing bugs present in the recipe presented above.
But, I'm not really here to talk about the recipe, or to impugn its author,
Michele Simionato, who has written several other excellent recipes on ASPN;
I have even personally used the DOT-grapher for inheritance hierarchies. I
doubt Michele spent much time on this quick hack, or considered it a
statement in the holy war I'm about to bring up, so please don't interpret
what follows as a personal attack.
What concerns me is that there is a persistent meme around the periphery of
the Twisted community that asynchronous programming is too hard, and that
things would be easier if it looked like it were multi-threaded. This
recently came up in a mailing
list post I wrote as well.
My personal opinion on this, and I believe this is a matter of public
record, is as follows: CONCURRENCY IS HARD. If
you are going to write concurrent programs you need to think about it all
the time; you need to plan for race conditions and draw your
state-transition diagrams and have big explicit comments in any section of
the code that has critical-section requirements even if you don't have to
"lock" it as with an event driven system. No inventions have really
significantly eased the cognitive difficulty of writing scalable concurrent
applications and it is unlikely that any will in the near term. Systems like
Twisted and Erlang have both provided powerful tools, but only if you are
smart and willing to invest energy in learning to use them properly; they
don't make the basic problems any easier. Most of all, threads
do not help, in fact, they make the problem worse in many cases. To
plagiarize a famous
Lisp fellow, if you have a concurrency problem, and you decide to use
threads, now you have two problems.
Let's put that aside for the moment, though.
Whether you agree with me about threads or not, though, Twisted was
written by, and is maintained by a large group of people who feel basically
the same way about this. We have some subtle
differences
about it, the consensus is the same. Threads are bad. Only use them when you
have to, and understand clearly what that means, don't loudly
provide "conveniences" for threads or use those "conveniences" for code
which could otherwise be written as non-blocking.
Please, Twisted users, please stop trying to turn Twisted into something it
isn't. If you want to use threads, write a multi-threaded program and please
stop trying to write infrastructure for Twisted to turn it into a big
multi-threaded application platform. WSGI and Zope efforts are excluded from
this comment, by the way: that's not trying to help people to write threaded
Twisted code, that's about trying to help Twisted be the container
for code written using a totally different paradigm, on a different
framework, and not written to directly use the Twisted
libraries.
Programs written with these kinds of thread-happy conveniences are generally
the ones which end up the buggiest, the hardest to test, and most likely the
least efficient as well. Worst of all, when you do run into those problems,
if you ask the Twisted dev team, you are likely to get a lot of smug "I told
you so", and very little actual help, since we have seen the problem before
and we keep trying to tell folks not to get started down this path.
Personally, It's frustrating to have that advice disregarded again and again
and still to get help requests from people who ignore it.
Imagine a man walks into a doctor's office, and says, "Doctor doctor, it
hurts when I do this OW", promptly shooting
himself in the hand with a nailgun. If this is the third time this week the
doctor has removed such a nail, do you think the doctor is going to show
this patient much patience? Now, imagine the doctor isn't getting paid for
his services. The fellow would be lucky to walk out without a second
nail...
You may have some awesome ideas about how multi-threaded programs should
work. Good for you. I love reading the work of people who think differently
than I do and succeed. It's a good way to learn. However, if you ask me, or
if you use Twisted, you are going to run into a lot of advice to discard
those ideas, a lot of roadblocks related to pervasive multi-threading, and
general "impedance mismatch" problems with the differences between the way
you think and the way Twisted works. It would probably be less work for you
to start from scratch, or to use a system that has threads as a fundamental
part of its programming model.
So, please, I'm not offended if you don't like Twisted, but if you like it,
appreciate it for what it is, and if you don't, don't bother with it at all.
Trying to use it while sweeping the most central parts of it under the rug
isn't going to help anyone, least of all you.