Panopticon Rift

Why exactly is it that Oculus Rift fans hate Facebook so much?

Greg Price expresses a common opinion here:

I myself had a similar reaction; despite not being particularly invested in VR specifically (I was not an Oculus backer) I felt pretty negatively about the fact that Facebook was the acquirer. I wasn’t quite sure why, at first, and after some reflection I’d like to share my thoughts with you on both why this is a bad thing and also why gamers, in particular, were disturbed by it.

The Oculus Rift really captured the imagination of the gaming community’s intelligentsia. John Carmack’s imprimatur alone was enough to get people interested, but the real power of the Oculus was to finally deliver on the promise of all those unbelievably clunky virtual reality headsets that we’ve played around with at one time or another.

Virtual Boy

The promise of Virtual Reality is, of course, to transport us so completely to another place and time that we cease to even be aware of the real world. It is that experience, that complete and overwhelming sense of being in an imagined place, that many gamers are looking for when they sit down in front of an existing game. It is the aspiration to that experience that makes “immersive” such a high compliment in game criticism.

Personally, immersion in a virtual world was the beginning of my real interest in computers. I’ve never been the kind of gamer who felt the need to be intensely competitive. I didn’t really care about rules and mechanics that much, at least not for their own sake. In fact, the term “gamer” is a bit of a misnomer - I’m more a connoisseur of interactive experiences. The very first “game” I remember really enjoying was Zork. Although Zork is a goal-directed game you can “win”, that didn’t interest me. Instead, I enjoyed wandering around the environments it provided, and experimenting with different actions to see what the computer would do.

Computer games are not really just one thing. The same term, “games”, encompasses wildly divergent experiences including computerized Solitaire, Silent Hill, Dyad, and Myst. Nevertheless, I think that pursuit of immersion – on really, fully, presently paying attention to an interactive experience – is a primary reason that “gamers” feel the need to distinguish themselves (ourselves?) from the casual computing public. Obviously, the fans of Oculus are among those most focused on immersive experiences.

Gamers feel the need to set themselves apart because computing today is practically defined by a torrential cascade of things that we’re only barely paying attention to. What makes computing “today” different than the computing of yesteryear is that computing used to be about thinking, and now it’s about communication.

The advent of social media has left us not only focused on communication, but focused on constant, ephemeral, shallow communication. This is not an accident. In our present economy there is, after all, no such thing as a “social media business” (or, indeed, a “search engine business”); there are only ad agencies.

The purveyors of social media need you to be engaged enough with your friends that you won’t leave their sites, so there is some level of entertainment or interest they must bring to their transactions with you, of course; but they don’t want you to be so engaged that a distracting advertisement would be obviously crass and inappropriate. Lighthearted banter, pictures of shiba inus, and shallow gossip are fantastic fodder for this. Less so are long, soul-searching long-form writing explaining and examining your feelings.

An ad for cat food might seem fine if you’re chuckling at a picture of a dog in a hoodie saying “wow, very meme, such meta”. It’s less likely to drive you through to the terminus of that purchase conversion funnel if you’re intently focused on supporting a friend who is explaining to you how a cancer scare drove home how they’re doing nothing with their life, or how your friend’s sibling has run away from home and your friend doesn’t know if they’re safe.

Even if you’re a highly social extrovert, the dominant emotion that this torrent of ephemeral communication produces is, at best, sarcastic amusement. More likely, it produces constant anxiety. We do not experience our better selves when we’re not really paying focused attention to anything1. As Community Season 5 Episode 8 recently put it, somewhat more bluntly: “Mark Zuckerberg is Fidel Castro in flip-flops.”

I think that despite all the other reasons we’re all annoyed - the feelings of betrayal around the Kickstarter, protestations of Facebook’s creepyness, and so on - the root of the anger around the Facebook acquisition is the sense that this technology with so much potential to reverse the Balkanization of our attention has now been put directly into the hands of those who created the problem in the first place.

So now, instead of looking forward to a technology that will allow us to visit a world of pure imagination, we now eagerly await something that will shove distracting notifications and annoying advertisements literally an inch away from our eyeballs. Probably after charging us several hundred dollars for the privilege.

It seems to me that’s a pretty clear justification for a few hundred negative reddit comments.


  1. Paid link. See disclosures

Unyielding

Be as the reed, not the oak tree. Green threads are just threads.

The Oak and the Reed by Achille Michallon

… that which is hard and stiff
is the follower of death
that which is soft and yielding
is the follower of life …

the Tao Te Ching, chapter 76

Problem: Threads Are Bad

As we know, threads are a bad idea, (for most purposes). Threads make local reasoning difficult, and local reasoning is perhaps the most important thing in software development.

With the word “threads”, I am referring to shared-state multithreading, despite the fact that there are languages, like Erlang and Haskell which refer to concurrent processes – those which do not implicitly share state, and require explicit coordination – as “threads”.

My experience is mainly (although not exclusively) with Python but the ideas presented here should generalize to most languages which have global shared mutable state by default, which is to say, quite a lot of them: C (including Original Recipe, Sharp, Extra Crispy, Objective, and Plus Plus), JavaScript, Java, Scheme, Ruby, and PHP, just to name a few.

With the phrase “local reasoning”, I’m referring to the ability to understand the behavior (and thereby, the correctness) of a routine by examining the routine itself rather than examining the entire system.

When you’re looking at a routine that manipulates some state, in a single-tasking, nonconcurrent system, you only have to imagine the state at the beginning of the routine, and the state at the end of the routine. To imagine the different states, you need only to read the routine and imagine executing its instructions in order from top to bottom. This means that the number of instructions you must consider is n, where n is the number of instructions in the routine. By contrast, in a system with arbitrary concurrent execution – one where multiple threads might concurrently execute this routine with the same state – you have to read the method in every possible order, making the complexity nn.

Therefore it is – literally – exponentially more difficult to reason about a routine that may be executed from an arbitrary number of threads concurrently. Instead, you need to consider every possible caller across your program, understanding what threads they might be invoked from, or what state they might share. If you’re writing a library desgined to be thread-safe, then you must place some of the burden of this understanding on your caller.

The importance of local reasoning really cannot be overstated. Computer programs are, at least for the time being, constructed by human beings who are thinking thoughts. Correct computer programs are constructed by human beings who can simultaneously think thoughts about all the interactions that the portion of the system they’re developing will have with other portions.

A human being can only think about seven things at once, plus or minus two. Therefore, although we may develop software systems that contain thousands, millions, or billions of components over time, we must be able to make changes to that system while only holding in mind an average of seven things. Really bad systems will make us concentrate on nine things and we will only be able to correctly change them when we’re at our absolute best. Really good systems will require us to concentrate on only five, and we might be able to write correct code for them even when we’re tired.

Aside: “Oh Come On They’re Not That Bad”

Those of you who actually use threads to write real software are probably objecting at this point. “Nobody would actually try to write free-threading code like this,” I can hear you complain, “Of course we’d use a lock or a queue to introduce some critical sections if we’re manipulating state.”

Mutexes can help mitigate this combinatorial explosion, but they can’t eliminate it, and they come with their own cost; you need to develop strategies to ensure consistent ordering of their acquisition. Mutexes should really be used to build queues, and to avoid deadlocks those queues should be non-blocking but eventually a system which communicates exclusively through non-blocking queues effectively becomes a set of communicating event loops, and its problems revert to those of an event-driven system; it doesn’t look like regular programming with threads any more.

But even if you build such a system, if you’re using a language like Python (or the ones detailed above) where modules, classes, and methods are all globally shared, mutable state, it’s always possible to make an error that will affect the behavior of your whole program without even realizing that you’re interacting with state at all. You have to have a level of vigilance bordering on paranoia just to make sure that your conventions around where state can be manipulated and by whom are honored, because when such an interaction causes a bug it’s nearly impossible to tell where it came from.

Of course, threads are just one source of inscrutable, brain-bending bugs, and quite often you can make workable assumptions that preclude you from actually having to square the complexity of every single routine that you touch; for one thing, many computations don’t require manipulating state at all, and you can (and must) ignore lots of things that can happen on every line of code anyway. (If you think not, when was the last time you audited your code base for correct behavior in the face of memory allocation failures?) So, in a sense, it’s possible to write real systems with threads that perform more or less correctly for the same reasons it’s possible to write any software approximating correctness at all; we all need a little strength of will and faith in our holy cause sometimes.

Nevertheless I still think it’s a bad idea to make things harder for ourselves if we can avoid it.

Solution: Don’t Use Threads

So now I’ve convinced you that if you’re programming in Python (or one of its moral equivalents with respect to concurrency and state) you shouldn’t use threads. Great. What are you going to do instead?

There’s a lot of debate over the best way to do “asynchronous” programming - that is to say, “not threads”, four options are often presented.

  1. Straight callbacks: Twisted’s IProtocol, JavaScript’s on<foo> idiom, where you give a callback to something which will call it later and then return control to something (usually a main loop) which will execute those callbacks,
  2. “Managed” callbacks, or Futures: Twisted’s Deferred, JavaScript’s Promises/A[+], E’s Promises, where you create a dedicated result-that-will-be-available-in-the-future object and return it for the caller to add callbacks to,
  3. Explicit coroutines: Twisted’s @inlineCallbacks, Tulip’s yield from coroutines, C#’s async/await, where you have a syntactic feature that explicitly suspends the current routine,
  4. and finally, implicit coroutines: Java’s “green threads”, Twisted’s Corotwine, eventlet, gevent, where any function may switch the entire stack of the current thread of control by calling a function which suspends it.

One of these things is not like the others; one of these things just doesn’t belong.

Don’t Use Those Threads Either

Options 1-3 are all ways of representing the cooperative transfer of control within a stateful system. They are a semantic improvement over threads. Callbacks, Futures, and Yield-based coroutines all allow for local reasoning about concurrent operations.

So why does option 4 even show up in this list?

Unfortunately, “asynchronous” systems have often been evangelized by emphasizing a somewhat dubious optimization which allows for a higher level of I/O-bound concurrency than with preemptive threads, rather than the problems with threading as a programming model that I’ve explained above. By characterizing “asynchronousness” in this way, it makes sense to lump all 4 choices together.

I’ve been guilty of this myself, especially in years past: saying that a system using Twisted is more efficient than one using an alternative approach using threads. In many cases that’s been true, but:

  1. the situation is almost always more complicated than that, when it comes to performance,
  2. “context switching” is rarely a bottleneck in real-world programs, and
  3. it’s a bit of a distraction from the much bigger advantage of event-driven programming, which is simply that it’s easier to write programs at scale, in both senses (that is, programs containing lots of code as well as programs which have many concurrent users).

A system that presents “implicit coroutines” – those which may transfer control to another concurrent task at any layer of the stack without any syntactic indication that this may happen – are simply the dubious optimization by itself.

Despite the fact that implicit coroutines masquerade under many different names, many of which don’t include the word “thread” – for example, “greenlets”, “coroutines”, “fibers”, “tasks” – green or lightweight threads are indeed threads, in that they present these same problems. In the long run, when you build a system that relies upon them, you eventually have all the pitfalls and dangers of full-blown preemptive threads. Which, as shown above, are bad.

When you look at the implementation of a potentially concurrent routine written using callbacks or yielding coroutines, you can visually see exactly where it might yield control, either to other routines, or perhaps even re-enter the same routine concurrently. If you are using callbacks – managed or otherwise – you will see a return statement, or the termination of a routine, which allows execution of the main loop to potentially continue. If you’re using explicit coroutines, you’ll see a yield (or await) statement which suspends the coroutine. Because you can see these indications of potential concurrency, they’re outside of your mind, in your text editor, and you don’t need to actively remember them as you’re working on them.

You can think of these explicit yield-points as places where your program may gracefully bend to the needs of concurrent inputs. Crumple zones, or relief valves, for your logic, if you will: a single point where you have to consider the implications of a transfer of control to other parts of your program, rather than a rigid routine which might transfer (break) at any point beyond your control.

Like crumple zones, you shouldn’t have too many of them, or they lose their effectiveness. A long routine which has an explicit yield point before every single instruction requires just as much out-of-order reasoning, and is therefore just as error-prone as one which has none, but might context switch before any instruction anyway. The advantage of having to actually insert the yield point explicitly is that at least you can see when a routine has this problem, and start to clean up and consolidate the mangement of its concurrency.

But this is all pretty abstract; let me give you a specific practical example, and a small theoretical demonstration.

The Buggiest Bug

Brass Cockroach - Image Credit GlamourGirlBeads http://www.etsy.com/listing/62042780/large-antiqued-brass-cockroach1-ants3074

When we wrote the very first version of Twisted Reality in Python, the version we had previously written in Java was already using green threads; at the time, the JVM didn’t have any other kind of threads. The advantage to the new networking layer that we developed was not some massive leap forward in performance (the software in question was a multiplayer text adventure, which at the absolute height of its popularity might have been played by 30 people simultaneously) but rather the dramatic reduction in the number and severity of horrible, un-traceable concurrency bugs. One, in particular, involved a brass, mechanical cockroach which would crawl around on a timer, leaping out of a player’s hands if it was in their inventory, moving between rooms if not. In the multithreaded version, the cockroach would leap out of your hands but then also still stay in your hands. As the cockroach moved between rooms it would create shadow copies of itself, slowly but inexorably creating a cockroach apocalypse as tens of thousands of pointers to the cockroach, each somehow acquiring their own timer, scuttled their way into every player’s inventory dozens of times.

Given that the feeling that this particular narrative feature was supposed to inspire was eccentric whimsy and not existential terror, the non-determinism introduced by threads was a serious problem. Our hope for the even-driven re-write was simply that we’d be able to diagnose the bug by single-stepping through a debugger; instead, the bug simply disappeared. (Echoes of this persist, in that you may rarely hear a particularly grizzled Twisted old-timer refer to a particularly intractable bug as a “brass cockroach”.)

The original source of the bug was so completely intractable that the only workable solution was to re-write the entire system from scratch. Months of debugging and testing and experimenting could still reproduce it only intermittently, and several “fixes” (read: random, desperate changes to the code) never resulted in anything.

I’d rather not do that ever again.

Ca(sh|che Coherent) Money

Despite the (I hope) entertaining nature of that anecdote, it still might be somewhat hard to visualize how concurrency results in a bug like that, and the code for that example is far too sprawling to be useful as an explanation. So here's a smaller in vitro example. Take my word for it that the source of the above bug was the result of many, many intersecting examples of the problem described below.

As it happens, this is the same variety of example Guido van Rossum gives when he describes why chose to use explicit coroutines instead of green threads for the upcoming standard library asyncio module, born out of the “tulip” project, so it's happened to more than one person in real life.

Photo Credit: Ennor https://www.flickr.com/photos/ennor/441394582/sizes/l/

Let’s say we have this program:

1
2
3
4
5
6
7
8
9
def transfer(amount, payer, payee, server):
    if not payer.sufficient_funds_for_withdrawl(amount):
        raise InsufficientFunds()
    log("{payer} has sufficient funds.", payer=payer)
    payee.deposit(amount)
    log("{payee} received payment", payee=payee)
    payer.withdraw(amount)
    log("{payer} made payment", payer=payer)
    server.update_balances([payer, payee])

(I realize that the ordering of operations is a bit odd in this example, but it makes the point easier to demonstrate, so please bear with me.)

In a world without concurrency, this is of course correct. If you run transfer twice in a row, the balance of both accounts is always correct. But if we were to run transfer with the same two accounts in an arbitrary number of threads simultaneously, it is (obviously, I hope) wrong. One thread could update a payer’s balance below the funds-sufficient threshold after the check to see if they’re sufficient, but before issuing the withdrawl.

So, let’s make it concurrent, in the PEP 3156 style. That update_balances routine looks like it probably has to do some network communication and block, so let’s consider that it is as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
@coroutine
def transfer(amount, payer, payee, server):
    if not payer.sufficient_funds_for_withdrawl(amount):
        raise InsufficientFunds()
    log("{payer} has sufficient funds.", payer=payer)
    payee.deposit(amount)
    log("{payee} received payment", payee=payee)
    payer.withdraw(amount)
    log("{payer} made payment", payer=payer)
    yield from server.update_balances([payer, payee])

So now we have a trivially concurrent, correct version of this routine, although we did have to update it a little. Regardless of what sufficient_funds_for_withdrawl, deposit and withdrawl do - even if they do network I/O - we know that we aren’t waiting for any of them to complete, so they can’t cause transfer to interfere with itself. For the sake of a brief example here, we’ll have to assume update_balances is a bit magical; for this to work our reads of the payer and payee’s balance must be consistent.

But if we were to use green threads as our “asynchronous” mechanism rather than coroutines and yields, we wouldn’t need to modify the program at all! Isn’t that better? And only update_balances blocks anyway, so isn’t it just as correct?

Sure: for now.

But now let’s make another, subtler code change: our hypothetical operations team has requested that we put all of our log messages into a networked log-gathering system for analysis. A reasonable request, so we alter the implementation of log to write to the network.

Now, what will we have to do to modify the green-threaded version of this code? Nothing! This is usually the point where fans of various green-threading systems will point and jeer, since once the logging system is modified to do its network I/O, you don’t even have to touch the code for the payments system. Separation of concerns! Less pointless busy-work! Looks like the green-threaded system is winning.

Oh well. Since I’m still a fan of explicit concurrency management, let’s do the clearly unnecessary busy-work of updating the ledger code.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
@coroutine
def transfer(amount, payer, payee, server):
    if not payer.sufficient_funds_for_withdrawl(amount):
        raise InsufficientFunds()
    yield from log("{payer} has sufficient funds.", payer=payer)
    payee.deposit(amount)
    yield from log("{payee} received payment", payee=payee)
    payer.withdraw(amount)
    yield from log("{payer} made payment", payer=payer)
    yield from server.update_balances([payer, payee])

Well okay, at least that wasn’t too hard, if somewhat tedious. Sigh. I guess we can go update all of the ledger’s callers now and update them too…

…wait a second.

In order to update this routine for a non-blocking version of log, we had to type a yield keyword between the sufficient_funds_for_withdrawl check and the withdraw call, between the deposit and the withdraw call, and between the withdraw and update_balances call. If we know a little about concurrency and a little about what this program is doing, we know that every one of those yield froms are a potential problem. If those log calls start to back up and block, a payer may have their account checked for sufficient funds, then funds could be deducted while a log message is going on, leaving them with a negative balance.

If we were in the middle of updating lots of code, we might have blindly added these yield keywords without noticing that mistake. I've certainly done that in the past, too. But just the mechanical act of typing these out is an opportunity to notice that something’s wrong, both now and later. Even if we get all the way through making the changes without realizing the problem, when we notice that balances are off, we can look only (reasoning locally!) at the transfer routine and realize, when we look at it, based on the presence of the yield from keywords, that there is something wrong with the transfer routine itself, regardless of the behavior of any of the things it’s calling.

In the process of making all these obviously broken modifications, another thought might occur to us: do we really need to wait before log messages are transmitted to the logging system before moving on with our application logic? The answer would almost always be “no”. A smart implementation of log could simply queue some outbound messages to the logging system, then discard if too many are buffered, removing any need for its caller to honor backpressure or slow down if the logging system can’t keep up. Consider the way syslog says “and N more” instead of logging certain messages repeatedly. That feature allows it to avoid filling up logs with repeated messages, and decreases the amount of stuff that needs to be buffered if writing the logs to disk is slow.

All the extra work you need to do when you update all the callers of log when you make it asynchronous is therefore a feature. Tedious as it may be, the asynchronousness of an individual function is, in fact, something that all of its callers must be aware of, just as they must be aware of its arguments and its return type.

In fact you are changing its return type: in Twisted, that return type would be Deferred, and in Tulip, that return type is a new flavor of generator. This new return type represents the new semantics that happen when you make a function start having concurrency implications.

Haskell does this as well, by embedding the IO monad in the return type of any function which needs to have side-effects. This is what certain people mean when they say Deferreds are a Monad.

The main difference between lightweight and heavyweight threads is that it is that, with rigorous application of strict principles like “never share any state unnecessarily”, and “always write tests for every routine at every point where it might suspend”, lightweight threads make it at least possible to write a program that will behave deterministically and correctly, assuming you understand it in its entirety. When you find a surprising bug in production, because a routine that is now suspending in a place it wasn’t before, it’s possible with a lightweight threading system to write a deterministic test that will exercise that code path. With heavyweight threads, any line could be the position of a context switch at any time, so it’s just not tractable to write tests for every possible order of execution.

However, with lightweight threads, you still can’t write a test to discover when a new yield point might be causing problems, so you're still always playing catch-up.

Although it’s possible to do this, it remains very challenging. As I described above, in languages like Python, Ruby, JavaScript, and PHP, even the code itself is shared, mutable state. Classes, types, functions, and namespaces are all shared, and all mutable. Libraries like object relational mappers commonly store state on classes.

No Shortcuts

Despite the great deal of badmouthing of threads above, my main purpose in writing this was not to convince you that threads are, in fact, bad. (Hopefully, you were convinced before you started reading this.) What I hope I’ve demonstrated is that if you agree with me that threading has problematic semantics, and is difficult to reason about, then there’s no particular advantage to using microthreads, beyond potentially optimizing your multithreaded code for a very specific I/O bound workload.

There are no shortcuts to making single-tasking code concurrent. It's just a hard problem, and some of that hard problem is reflected in the difficulty of typing a bunch of new concurrency-specific code.

So don’t be fooled: a thread is a thread regardless of its color. If you want your program to be supple and resilient in the face of concurrency, when the storm of concurrency blows, allow it to change. Tell it to yield, just like the reed. Otherwise, just like the steadfast and unchanging oak tree in the storm, your steadfast and unchanging algorithms will break right in half.

And Now For Something Completely Different

I’ve switched to a new publishing platform. You no longer need to inform state security of your interest in this content.

It seems that the constant reminders of all the wonderful new features that my previous web publishing host had in store for me finally motivated me to set up a somewhat more do-it-yourself publishing arrangement, as behooves someone of my particular talents and skills.

Desk

I’m using Pelican now, and from what I’ve seen of it so far, it’s a promising tool. I particularly like their approach to publishing multiple content types; for content that I’m writing new, I can use markdown, but for content that I want to process automatically, it accepts HTML just fine, which means that the conversion process from my previous host has been relatively painless. (Relatively painless, that is. Remember to calibrate pain on the Extract-Transform-Load scale, which starts at 8 or so.)

One interesting consequence of this change is that you may now access this blog securely if that’s the sort of thing you are interested in.

Another change is that I’ve made the conscious decision to eliminate the comments section. While over the years I have been lucky to receive many interesting and engaging comments on my posts, I think that the best feedback I've gotten has always been from people writing in their own spaces. So I've mainly removed the temptation of the comment box to prevent the motivation to write something thougthful from being burned away in a quick riposte.

If you would like to comment on something I’ve said here, now or in the future, just send me an email and I’ll be happy to either post your comments inline, in a new post, or ideally, link to your own published response elsewhere.

Or, of course, ignore it entirely, as is my right as the lord of this particular digital fiefdom.

Sadly, this change makes some of the older posts imported to this blog read slightly oddly, as they invite you to “comment below”, but nobody’s been doing that for some time, so I think that overall this change is likely to provoke a better quality of discussion and use of time. The full indictment of how You Kids Today ruined the Internet with Your Horrible Comments, and Back In My Day Things Were Better And So Forth will have to wait for a future article. (Or you could go read any New York Times article with the word “millennials” in the title, as it will say basically the same thing.)

I hope you enjoy the new format.

On "On Women In Tech"

I just read the inimitable Lea Verou's "On Women In Tech", and I have a few thoughts.

(Hello, Internet.  Please don't leave ten thousand horrible comments on this post, or on hers.  Please, just this once.)

First, I should say that this essay declined to name any specific efforts.  While that is nice in that it avoids name-calling, it does make it a bit more challenging to construct a specific argument.  So, I apologize for the over-general nature of some of my arguments here, but perhaps it's for the best so we don't get involved in mud-slinging over particular personalities or particular groups.  However, I do have to say that there are of course some groups that are more effective than others, and not everything I say applies uniformly.  But I do believe it's generally true.

I really wanted to like Lea's piece.  As someone who is (trying to be, at least) actively involved with outreach efforts, I feel like sometimes those efforts can have a very self-congratulatory vibe, and they could use a bit of honest criticism from within.  This is especially important because much of the "criticism" of feminism comes from … well, let's say, "unsympathetic sources", since I'm sure that if I'm more specific than that I'll activate a particularly unpleasant sort of internet hate machine.

As Lea herself acknowledges, there is a real problem with a lack of women in the software industry.  Women are still hugely underrepresented.  Women-in-technology groups propose a variety of different solutions to this problem.  As someone attacking a solution to a real problem, I feel like Lea faces an additional burden of proof than someone arguing in favor of such a solution; but her criticism falls short in a very important way: a lack of data.

Generally speaking, women-in-technology groups gather all kinds of statistics on both the problem as well as the efficacy of various solutions.  And make no mistake: many of the proposed solutions that Lea doesn't like – women-only or women-preferred events and groups, increasing the visibility of women in leadership roles, and both banning sexualized content from conference presentations and publicly communicating that it has been banned – do work. Whenever I've spoken with my friends who fit (to greater or lesser degrees) into the stereotype that Lea paints of a women-in-technology activist, they know what they're talking about.  Moreover, in the context of Python groups of various stripes, they have the statistics to prove that the things they are trying to do work.  It's important to keep in mind that each of these activities addresses a different audience; not all women are the same, and different sub-groups need different things to get involved.

To Lea, and to other women who have read her post and immediately identify with it, one thing that you may want to consider is that these efforts are not about you. As a woman already excelling in the field of software technology, attracting your interest and participation is not as important to these groups as attracting new people: if you want to increase participation, you must, as a simple fact of arithmetic, get those who are not already participating to participate.  This may mean women in the field who just don't participate in the community for some reason, or it may mean getting women into the field who aren't in it at all.  That means those women won't be like you in some important characteristics; for example, in aggregate, women do identify with their gender more strongly than men do.

Despite her distaste for such efforts, I first came to follow Lea on twitter explicitly because of a "women in technology" effort.  I followed her after making a conscious decision to diversify my (unfortunately still mostly male, unfortunately still overwhelmingly white) twitter stream.  As a result of that decision, I found a list of prominent women in web technology, where I came across her twitter handle.

Am I continuing to be her follower today just because she's a woman?  No, of course not.  She's an entertaining character with a lot of very interesting stuff to say about a variety of technology - browser front-end issues, mostly centering on CSS - that I don't know a lot about.  As a result of her tweets I've read several of her presentations and I'm much better informed as a result.  This knowledge has been useful to me both professionally and personally.  (And, maybe, if she doesn't completely hate this post, it will have benefited Lea as well.)

This is just one example of a general pattern - if, when we notice a glaring lack of diversity, we make an effort to seek out and include members of an underrepresented group, we frequently find that they have something just as interesting to contribute as the existing over-represented group, if not more so. By simple virtue of being different on one axis, we often find that they are different on other axes as well, and therefore have a more interesting point of view to contribute.

(I feel the need to stress that this is not to say that "all women are X, and therefore if we get more women in technology we will get more much-needed X".  That's reductive and probably factually inaccurate for any given X you might select.  The point is that if you pay attention to lots of people you haven't previously been paying attention to, you will, almost by definition, learn new things.)

The point is, I did not follow Lea because I had a "quota".  She did not claim a spot on my twitter stream that was previously occupied by a better-qualified man.  I just made an effort to follow more women and discovered that there were many interesting people I'd somehow missed out on.  And this worked out quite well for me as now I follow more interesting people.

Wherever efforts like this are made by institutional groups - conferences, for example - to ask an underrepresented group to participate more, to give a second look to applications from that underrepresented group - that accusation, "quotas", always tends to quickly follow.  Usually it comes from members of the over-represented group who didn't make the cut, complaining that they're being excluded despite being "more qualified", but the fact that Lea is herself a woman doesn't make her claim about "quotas" any more true.

Generally, the attempt to include female speakers in a conference program is not enforced via a "quota", unspoken or not; there are usually more than enough qualified female speakers, who are, for one reason or another, either (A) not applying, or (B) being rejected by a flawed "objective" selection process.  The fact is that it is very, very hard to tell whether a speaker will give a good talk or not, even subjectively, in advance.  There is basically no objective metric, so we can't say that we currently have a pure meritocracy, since we can't even agree on what merit is, for conference speakers.

Additionally, myriad cognitive biases influence our judgment.  For example, in an often-repeated story, blind auditions radically (50%) improve the chances that a woman will be selected in an orchestra.  Is this because all conductors are knuckle-dragging misogynists?  Well, okay, some are, but for the most part, that's probably not the reason: it's just that we – both genders in many cultures – are primed, since childhood, to regard woman as less capable, and repeated scientific analysis has shown that that bias creeps into our thinking in all kinds of contexts.

(An aside to the dudes in the audience: if you want to be an ally to feminists, the thing to do is never to say "well I'm not a sexist, I would never let gender influence my judgment".  It's to say "I know my judgment might be compromised in ways I can't control, so I'm going to take steps to ensure that doesn't negatively affect anyone".  A good general rule for privileged classes of all types.)

However, even if women were, for some reason, less qualified, and an actual quota were needed to get women on stage at technical conferences, it would still be worth it.  Let me reiterate first though: the women who apply for talks are not generally less qualified, and such quotas are generally not necessary.  One reason that you really don't need a quota for this particular role is that experience is not really a factor in giving a good talk; in fact, it can work against you.

Both of my two favorite talks from a recent technical conference I attended (sorry, no links, since I'm not going to subject the speakers to any fallout from this article, if there is any, but I'm happy to give you a link if you get in touch) were by women who were relatively new to the community about being relatively new to the community.  Of course many members of the old boys' club gave great talks too (myself included, or so I'd like to think) but there's also always a few talks we've heard before, and always at least a few where the speaker just reads the bullets on their slides about some boring thing like the open source YAML parser that they wrote, while standing motionless and staring down at the podium.

The reason that even quotas would still be worth it, if they were necessary, is that, for the next generation of potential hackers, it is very important that women in roles of technical leadership be seen as normal.  Children very quickly pick up on social cues; this is when they are establishing all those pernicious cognitive biases that I mentioned previously.  Lea herself points this out, saying "If you don’t meet many technical women, your brain tends to pick up the pattern".  In fact the entire final section of her article is about "starting early", and addressing young girls rather than adult women.  It's worth noting that stereotypical "women in technology" programs are, of course, already doing this; it's not an either-or proposition.  But in order to convince young women and girls that this is a normal, sensible thing to do, they need to be able to visualize themselves being successful, and that means there need to be visible, successful older women in the industry.

Finally, let's examine a claim that Lea makes:

"It’s not our industry that has a sexism problem, our society has a sexism problem."

While our society clearly does have a sexism problem, it's possible that our industry does in fact especially have a sexism problem.  So, does it?  Let's go to the numbers; the United States Department of Labor's Bureau of Labor Statistics maintains this handy list: "Employed persons by occupation, sex, and age".

According to that list, the total number of employed men in the USA over 20 is 73,403,000, and the total number of employed women over 20 is 64,640,000.  That means 53% of the workforce is male, which is a 3% bias (disregarding, for the moment, issues of pay equity).

Now let's look at "Computer and mathematical occupations".  The number of men in those professions, 20 and over, is 2,834,000.  The number of women in that same group of occupations, in that same age range, is 972,000.  That means that 74% of this workforce is men, which is a 24% bias, or eight times the bias for general employment.  So: yes, our industry, in particular, has a sexism problem.  But, perhaps comparing against general employment is unfair; let's look instead at management, a famously unfavorable field for women, which sports 9,823,000 over-19 men and 6,167,000 over-19 women, or 61% men, or an 11% bias.  In other words, computer technology is over twice as hostile to women in the USA, statistically speaking, than the historically unfair field of management.

In closing, I'd like to say that, despite some obviously substantial disagreement, I think that Lea has some good insight into aspects of the unintentional negative consequences that some of the strategies that outreach programs for women in technology have.  I hope that in the future, she'll return to the topic without resorting to the tired clichés about quotas, political correctness, "it's society, it's not our industry", and falsely equating the consequences of male and female sexual objectification.  There are still many good unanswered questions that she raises, like:

  • How can women avoid being seen as unduly concerned with things like language and propriety, while still drawing a firm line around unacceptable behavior?
  • How can conference organizers ensure that they're not overzealously including talks from women that aren't up to their usual quality standards?
  • What should we do about the perception that women are being unfairly selected when, in fact, this is almost never the case?
  • How can we get the message out to men who are unused to seeing women at their professional events that loudly apologizing only to the only woman in the room for using profanity is probably worse than using profanity in the first place?
  • How can the women-in-technology movement avoid alienating women who are interested in technology for technology's sake, and explicitly because they don't want to be involved with annoying social stuff like being an activist?  What's a good protocol for identifying that way without being seen to repudiate feminism as a whole?
  • How can we get naive but well-meaning men to stop treating all women as ambassadors for their gender, with all the baggage that implies?
I hope that in the future, Lea, and others who agree with her, will take some time to dig deeper into the realities of the women-in-tech movement, and perhaps work with those already in said movement to provide better answers to these questions.  It seems to me that there's more agreement than disagreement here, and that much of what she dislikes is a caricature of the movement, not its essence.

The Twisted Way

One of the things that confuses me most about Twisted is the fact that so many people seem to be confused by things about Twisted.

Much has been written, some of it by me, some of it by other brilliant members of the community, attempting to explain Twisted in lots of detail so that you can use it and understand it and control it to do your bidding.  But today, I'd like to try something different, and instead of trying to help you figure out how to use Twisted, I will try to help you understand what Twisted is.  To aid you in meditating upon its essence and to understand how it is a metaphor for software, and, if you are truly enlightened, for all life.

Let us contemplate the Twisted Way.

Image Credit: Ian Sane

In the beginning was the Tao.
All things issue from it; all things return to it.
- Tao Te Ching

All information systems are metaphors for the world.  All programs are, at least in small part, systems.  Therefore, every program contains within it a world of thought.  Those thoughts must be developed before they can be shared; therefore, one must have an interesting program before one thinks to do any interesting I/O.

It is less fashionable these days to speak of "object-oriented modeling" than it once was, but mostly because object-oriented design is now so pervasive that no-one needs convincing any more.  Nevertheless, that is what almost all of us do.  When an archetypical programmer in this new millennium sets out to create a program, they will typically begin by creating a Class, and then endowing that Class with Behavior; then, by creating an Instance of that Class, and bestowing interesting Data upon that Instance.

Such an Instance receives (as input) method calls from some other object, and produces (as output) method calls on some other object.  It is a system unto itself, simulating some aspect of human endeavor, computing useful results and dispensing them, all in an abstract vacuum.

But, the programmers who produced this artifact desires it to interact with the world; to produce an effect, and therefore to accept Input and dispense Output.

It is at this point that the programmer encounters Twisted.

When you look for it, there is nothing to see.
When you listen for it, there is nothing to hear.
When you use it, it is inexhaustible.

Except that, in fact, nobody ever encounters Twisted this way.  If this is where – and how – you encounter Twisted, then you will likely have great success with it.  But everyone tends to encounter Twisted, like one encounters almost every other piece of infrastructure, in medias res.  Method calls are flying around all over the place in some huge inscrutable system and you just have to bang through the tutorial to figure it all out right now, and it looks super weird.

Over the years, so many questions I've answered about Twisted seem to reduce to: "how do I even get this thing to do anything"?

This is Twisted's great mystery: it does nothing.  By itself, it is the world's largest NOP.  Its job, purely and simply is to connect your object to the world.  You tell Twisted: listen for connections on this port; when one is made, do this.  Make this request, and when it has a response, do that.  Listen for email over SMTP; when one arrives, do the other.

Without your direction, reactor.run will just ... wait.

The source of most confusion with Twisted, I believe, is that few objects are designed in this idiom.  When we seek to create a program, we feel we must start interacting with it immediately, before it even knows what it is supposed to do.  The seductions of blocking I/O are many and varied.  Any function which appears to merely compute a result can simply be changed to cheat and get its answer by asking some other system instead, with its callers none the wiser.  Even for those of us who know better, these little cheats accumulate and make the program brittle and slow, and force it to be spun out into a thread, or the cold, sparse desert of its own separate process, so it may tediously plod along, waiting for the response to its each and every query.

Thus, desire (for immediate I/O) leads to suffering (of the maintenance programmer).

Return is the movement of the Tao.
Yielding is the way of the Tao.

It doesn't need to be that way, though.  When you create an object, it is best to create it as independently as possible; to test it in isolation; to discretely separate its every interaction with the outside world so that they may be carefully controlled, monitored, intercepted and inspected, one iteration at a time.

All your object needs to do is to define its units of work as atomic, individual functions that it wishes to perform; then, return to its caller and allow it to proceed.

The Master does their job and then stops.
They understand that the universe is forever out of control.

When you design your objects by contemplating their purpose and making them have a consistent, nicely separated internal model of what they're supposed to represent, Twisted seems less like a straightjacket, contorting your program into some awkward shape.  Instead, it becomes a comfortable jacket that your object might slip on to protect itself from the vicissitudes of whatever events may assault it from the network, whether they be the soft staccato of DNS, the confused warbling of SIP or the terrifying roar of IMAP.

Better yet, your object can be a model of some problem domain, and will therefore have a dedicated partner; a different object, a wrapper, whose entire purpose is to translate from a lexicon of network-based events, timers, and Deferred callbacks, into a language that is more directly applicable to your problem domain.  After all, each request or response from the network means something to your application, otherwise it would not have been made; the process of explicitly enumerating all those meanings and recording and documenting them in a module dedicated to that purpose is a very useful exercise.

When your object has such unity of purpose and clarity of function, then Twisted can help manage its stream of events even if the events are not actually coming from a network; abstractions like deferreds, producers, consumers, cooperators and inline callbacks can be used to manipulate timers, keystrokes, and button clicks just as easily as network traffic.

True words aren't eloquent; eloquent words aren't true.
Sages don't need to prove their point;
those who need to prove their point aren't wise.

So, if you are setting out to learn to use Twisted, approach it in this manner: it is not something that will, itself, give your object's inner thoughts structure, purpose and meaning.  It is merely a wrapper; an interstitial layer between your logic and some other system.  The methods it calls upon you might be coming from anywhere.  And indeed, they should be coming from at least one other place: your unit tests.

(With apologies to Geoffrey James and Laozi.)