Twisted was originally the network component of a text-based role-playing game engine. In the late 90s when it was first developed, the idea of a video game being able to send you actual emails and instant messages1 seemed pretty neat, rather than the valence that it would have today (nightmare). We never ended up making an actual game, so we eventually removed the “game engine” piece, and just kept the protocol implementations of the email and the AOL Instant Messenger, and the DNS and eventually SSH and so on. The residual pieces of that project have not seen maintenance for a decade.
So it’s not really a way to make a text-based game any more.
Then, Twisted was the premier longest-lived and best-maintained event-driven I/O library for Python. If you wanted to make high-performance servers, it was the best and most obvious way to do it. Even if it still acquits itself relatively well in this regard, its claim to fame, and to indispensability, has been somewhat obviated by the inclusion of asyncio in the standard library. Even if it has some stuff to recommend it — better-than-average testing facilities, for example — you can get better-documented and better-integrated low-level asynchronous networking tools without even installing any packages.
So it’s not really the premier choice for an event-loop these days.
Twisted does still contain an SSH server and client, an email server and client, a DNS server and client, a web server and client (now including websockets). But you can cobble these together by installing a variety of packages from other sources.
So... it’s a bunch of protocols? And I guess it doesn’t use AI? But... why would anyone care about that? Why contribute to such a thing?
What is Twisted doing, and who is it doing it for? What is the political project of Twisted, nigh on 30 years into its existence?
Most of all, why should you care about it, or want to contribute?
If you’re asking what you could use Twisted for today, it’s an excellent choice if you’re implementing a new network service and you need more than one protocol. If you need to do HTTP, DNS, packet capture, and a custom SSH server, it would be very hard to arrange a collage of other Python packages to manage this. It’s still much better at deterministic testing than most of the standard library. It’s a bit easier to hook into a GUI.
But that’s not what Twisted is doing.
What I want Twisted to do for the world is still an unfinished project. It’s honestly kind of bad at doing what I want it to, but it could be great. The overarching goal of having and maintaining all this stuff is to, as much as possible, demystify networked software operation and let you build Internet facing systems at a human scale.
It’s all in a memory-safe language with good error handling, so you can avoid
the scariest security pitfalls. It’s all integrated into a single paradigm
with built-in tools for testing. It’s designed to scale down to be runnable
on the average laptop, and not require large elaborate server farms or
distributed clusters to run. Via the twist command line utility, it provides
a Swiss army knife of simple tools that you can use out of the box.
In a world where all of our compute is in massive distant opaque datacenters, where a tiny cartel of gigantic tech companies control nearly the whole supply chain, one of the few things that you still can have agency over is your own server on the Internet.
It’s still possible to get a domain name for a few bucks a year, a free-tier or very low-cost VPS, then stick a glue record in and run your own DNS. From that point forward, you can run your own email, your own web server, your own terminal server. But doing this requires deep expertise, and that expertise requirement is massively inflated by the operational concerns that go with running twenty different pieces of software from twenty different projects, probably in twenty different languages, with completely inconsistent aesthetics.
The reason that Twisted is event-driven is because as much as it can be a pain
to wrap your head around async/await (or worse yet, Deferred directly),
it is much easier for a single person to understand the concurrency
requirements of an async I/O multiplexed system than to understand what is
going on with an application that requires multiple thread pools. It is
possible to understand concurrent code that manages parallel tasks in that
idiom. The reason that Twisted integrates so many different protocol
implementations and attempts2 to unify them with a common authentication
layer is to reduce
the overhead required to manage a comprehensive suite of network services.
Even the not-using-AI thing is not an aside due to some accidental personal preference about tools — it’s a core part of building an engine that you can personally understand if you dive into it, that’s built with a consistent style and layered in a way that you can take apart if you need to. It’s not built with AI because we’re not building for AI, we’re building for people and the entire point is that people are supposed to be able to understand it.
We used to have a tag line, which is somewhat less emphasized these days because its meaning is not immediately clear without this background, but was once quite prominently featured on the home page: “Twisted: Building the Engine of Your Internet”. That’s what I want to bring to you: an engine for you to build your Internet, that is small enough that you can make it personal, that doesn’t require massive investment in management tooling just to get started.
Now, for various reasons, I no longer use twisted.names to host my own DNS
(although I did for many years!), and we never wrote enough of a twisted.mail
suite for it to be usable for email hosting on its own. But, if you’re reading
this article, it was probably delivered to you directly via Twisted’s own
twisted.web HTTP server, using twisted.protocols.tls to configure our own
OpenSSL TLS termination. I have run with extremely high reliability for years
and next to zero management overhead, in a single simple docker container.
Which I then administer and upload files to via Twisted’s own twisted.conch
SSH server.
If that seems like an appealing project and you’d like to volunteer to help out, drop me a line and maybe I can point you at some tasks to get started. Twisted falls far short of this aspiration despite 30 years of work, and we could use all the help we can get.
Acknowledgments
Thank you to my patrons who are supporting my writing on this blog. If you like what you’ve read here and you’d like to read more of it, or you’d like to support my various open-source endeavors, you can support my work as a sponsor!