This post recommends calling pygame.display.flip from a thread, which I tested extensively on mac, windows, and linux before posting, but after some feedback from readers, I realize that this strategy is not in fact cross-platform; specifically, the nvidia drivers on linux appear to either crash or display a black window if you try to do this. The SDL FAQ does say that you can’t call “video functions” from multiple threads, and flip does do that under the hood. I do plan to update this post again, either with a method to make it safe, or a method to use slightly more complex timing heuristics to accomplish the same thing. In the meanwhile, please be aware that this may cause portability problems for your code.

I’ve written about this before, but in that context I was writing mainly about frame-rate independence, and only gave a brief mention of vertical sync; the title also mentioned Twisted, and upon re-reading it I realized that many folks who might get a lot of use out of its technique would not have bothered to read it, just because I made it sound like an aside in the context of an animation technique in a game that already wanted to use Twisted for some reason, rather than a comprehensive best practice. Now that Pygame 2.0 is out, though, and the vsync=1 flag is more reliably available to everyone, I thought it would be worth revisiting.

Per the many tutorials out there, including the official one, most Pygame mainloops look like this:

1
2
3
4
5
6
7
8

pygame.display.set_mode((320, 240))

while 1:
    for event in pygame.event.get():
        handleEvent(event)
    for drawable in myDrawables:
        drawable.draw()
    pygame.display.flip()

Obviously that works okay, or folks wouldn’t do it, but it can give an impression of a certain lack of polish for most beginner Pygame games.

The thing that’s always bothered me personally about this idiom is: where does the networking go? After spending many years trying to popularize event loops in Python, I’m sad to see people implementing loops over and over again that have no way to get networking, or threads, or timers scheduled in a standard way so that libraries could be written without the application needing to manually call them every frame.

But, who cares how I feel about it? Lots of games don’t have networking¹. There are more general problems with it. Specifically, it is likely to:

1. waste power, and
2. look bad.

Wasting Power

Why should anyone care about power when they’re making a video game? Aren’t games supposed to just gobble up CPUs and GPUs for breakfast, burning up as much power as they need for the most gamer experience possible?

Chances are, if you’re making a game that you expect anyone that you don’t personally know to play, they’re going to be playing it on a laptop². Pygame might have a reputation for being “slow”, but for a simple 2D game with only a few sprites, Python can easily render several thousand frames per second. Even the fastest display in the world can only refresh at 360Hz³. That’s less than one thousand frames per second. The average laptop display is going to be more like 60Hz, or — if you’re lucky — maybe 120. By rendering thousands of frames that the user never even sees, you warm up their CPU uncomfortably⁴, and you waste 10x (or more) of their battery doing useless work.

At some point your game might have enough stuff going on that it will run the CPU at full tilt, and if it does, that’s probably fine; at least then you’ll be using up that heat and battery life in order to make their computer do something useful. But even if it is, it’s probably not doing that all of the time, and battery is definitely a use-over-time sort of problem.

Looking Bad

If you’re rendering directly to the screen without regard for vsync, your players are going to experience Screen Tearing, where the screen is in the middle of updating while you’re in the middle of drawing to it. This looks especially bad if your game is panning over a background, which is a very likely scenario for the usual genre of 2D Pygame game.

How to fix it?

Pygame lets you turn on VSync, and in Pygame 2, you can do this simply by passing the pygame.SCALED flag and the vsync=1 argument to set_mode().

Now your game will have silky smooth animations and scrolling⁵! Solved!

But... if the fix is so simple, why doesn’t everybody — including, notably, the official documentation — recommend doing this?

The solution creates another problem: pygame.display.flip may now block until the next display refresh, which may be many milliseconds.

Even worse: note the word “may”. Unfortunately, behavior of vsync is quite inconsistent between platforms and drivers, so for a properly cross-platform game it may be necessary to allow the user to select a frame rate and wait on an asyncio.sleep than running flip in a thread. Using the techniques from the answers to this stack overflow answer you can establish a reasonable heuristic for the refresh rate of the relevant display, but if adding those libraries and writing that code is too complex, “60” is probably a good enough value to start with, even if the user’s monitor can go a little faster. This might save a little power even in the case where you can rely on flip to tell you when the monitor is actually ready again; if your game can only reliably render 60FPS anyway because there’s too much Python game logic going on to consistently go faster, it’s better to achieve a consistent but lower framerate than to be faster but inconsistent.

The potential for blocking needs to be dealt with though, and it has several knock-on effects.

For one thing, it makes my “where do you put the networking” problem even worse: most networking frameworks expect to be able to send more than one packet every 16 milliseconds.

More pressingly for most Pygame users, however, it creates a minor performance headache. You now spend a bunch of time blocked in the now-blocking flip call, wasting precious milliseconds that you could be using to do stuff unrelated to drawing, like handling user input, updating animations, running AI, and so on.

The problem is that your Pygame mainloop has 3 jobs:

1. drawing
2. game logic (AI and so on)
3. input handling

What you want to do to ensure the smoothest possible frame rate is to draw everything as fast as you possibly can at the beginning of the frame and then call flip immediately to be sure that the graphics have been delivered to the screen and they don’t have to wait until the next screen-refresh. However, this is at odds with the need to get as much done as possible before you call flip and possibly block for 1/60th of a second.

So either you put off calling flip, potentially risking a dropped frame if your AI is a little slow, or you call flip too eagerly and waste a bunch of time waiting around for the display to refresh. This is especially true of things like animations, which you can’t update before drawing, because you have to draw this frame before you worry about the next one, but waiting until after flip wastes valuable time; by the time you are starting your next frame draw, you possibly have other code which now needs to run, and you’re racing to get it done before that next flip call.

Now, if your Python game logic is actually saturating your CPU — which is not hard to do — you’ll drop frames no matter what. But there are a lot of marginal cases where you’ve mostly got enough CPU to do what you need to without dropping frames, and it can be a lot of overhead to constantly check the clock to see if you have enough frame budget left to do one more work item before the frame deadline - or, for that matter, to maintain a workable heuristic for exactly when that frame deadline will be.

The technique to avoid these problems is deceptively simple, and in fact it was covered with the deferToThread trick presented in my earlier post. But again, we’re not here to talk about Twisted. So let’s do this the no-additional-dependencies, stdlib-only way, with asyncio:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

import asyncio
import time
from math import inf

from pygame.display import set_mode, flip
from pygame.constants import SCALED
from pygame.event import get

event_handler = ...
drawables = [...]

async def pygame_loop(framerate_limit=inf):
    loop = asyncio.get_event_loop()
    screen_surface = set_mode(size=(480, 255), flags=SCALED, vsync=1)
    next_frame_target = 0.0
    limit_frame_duration = (1.0 / framerate_limit)

    while True:

        if limit_frame_duration:
            # framerate limiter
            this_frame = time.time()
            delay = next_frame_target - this_frame
            if delay > 0:
                await asyncio.sleep(delay)
            next_frame_target = this_frame + limit_frame_duration

        for drawable in drawables:
            drawable.draw(screen_surface)
        events_to_handle = list(get())
        events_handled = loop.create_task(handle_events(events_to_handle))
        await loop.run_in_executor(None, flip)
        # don’t want to accidentally start drawing again until events are done
        await events_handled

async def handle_events(events_to_handle):
    # note that this must be an async def even if it doesn’t await
    for event in events_to_handle:
        event_handler.handle_event(event)

asyncio.run(pygame_loop(120))

Go Forth and Loop Better

At some point I will probably release my own wrapper library⁶ which does something similar to this, but I really wanted to present this as a technique rather than as some packaged-up code to use, since do-it-yourself mainloops, and keeping dependencies to a minimum, are such staples of Pygame community culture.

As you can see, this technique is only a few lines longer than the standard recipe for a Pygame main loop, but you now have access to a ton of additional functionality:

• You can manage your framerate independence in both animations and game logic by just setting some timers and letting the frames update at the appropriate times; stop worrying about doing math on the clock by yourself!
• Do you want to add networked multiplayer? No problem! Networking all happens inside the event loop, make whatever network requests you want, and never worry about blocking the game’s drawing on a network request!
• Now your players’ laptops run cool while playing, and the graphics don’t have ugly tearing artifacts any more!

I really hope that this sees broader adoption so that the description “indie game made in Python” will no longer imply “runs hot and tears a lot when the screen is panning”. I’m also definitely curious to hear from readers, so please let me know if you end up using this technique to good effect!⁷

The essence of software engineering is solving problems.

The first impression of this insight will almost certainly be that it seems like a good thing. If you have a problem, then solving it is great!

But software engineers are more likely to have mental health problems¹ than those who perform mechanical labor, and I think our problem-oriented world-view has something to do with that.

So, how could solving problems be a problem?

As an example, let’s consider the idea of a bug tracker.

For many years, in the field of software, any system used to track work has been commonly referred to as a “bug tracker”. In recent years, the labels have become more euphemistic and general, and we might now call them “issue trackers”. We have Sapir-Whorfed² our way into the default assumption that any work that might need performing is a degenerate case of a problem.

We can contrast this with other fields. Any industry will need to track work that must be done. For example, in doing some light research for this post, I discovered that the relevant term of art in construction³ is typically “Project Management” or “Task Management” software. “Projects” and “Tasks” are no less hard work, but the terms do have a different valence than “Bugs” and “Issues”.

I don’t think we can start to fix this ... problem ... by attempting to change the terminology. Firstly, the domain inherently lends itself to this sort of language, which is why it emerged in the first place.

Secondly, Atlassian has desperately been trying to get everybody to call their bug tracker a “software development tool” where you write “stories” for years, and nobody does. It’s an issue tracker where you file bugs, and that’s what everyone calls it and describes what they do with it. Even they have to protest, perhaps a bit too much, that it’s “way more than a bug and issue tracker”⁴.

This pervasive orientation towards “problems” as the atom of work does extend to any knowledge work, and thereby to any “productivity system”. Any to-do list is, at its core, a list of problems. You wouldn’t put an item on the list if you were happy with the way the world was. Therefore every unfinished item in any to-do list is a little pebble of worry.

As of this writing, I have almost 1000 unfinished tasks on my personal to-do list.

This is to say nothing of any tasks I have to perform at work, not to mention the implicit א‎₀ of additional unfinished tasks once one considers open source issue trackers for projects I work on.

It’s not really reasonable to opt out of this habit of problematizing everything. This monument to human folly that I’ve meticulously constructed out of the records of aspirations which exceed my capacity is, in fact, also an excellent prioritization tool. If you’re a good engineer, or even just good at making to-do lists, you’ll inevitably make huge lists of problems. On some level, this is what it means to set an intention to make the world — or at least your world — better.

On a different level though, this is how you set out to systematically give yourself anxiety, depression, or both. It’s clear from a wealth of neurological research that repeated experiences and thoughts change neural structures⁵. Thinking the same thought over and over literally re-wires your brain. Thinking the thought “here is another problem” over and over again forever is bound to cause some problems of its own.

The structure of to-do apps, bug trackers and the like is such that when an item is completed — when a problem is solved — it is subsequently removed from both physical view and our mind’s eye. What would be the point of simply lingering on a completed task? All the useful work is, after all, problems that haven’t been solved yet. Therefore the vast majority of our time is spent contemplating nothing but problems, prompting the continuous potentiation⁶ of neural pathways which lead to despair.

I don’t want to pretend that I have a cure for this self-inflicted ailment. I do, however, have a humble suggestion for one way to push back just a little bit against the relentless, unending tide of problems slowly eroding the shores of our souls: a positivity journal.

By “journal”, I do mean a private journal. Public expressions of positivity⁷ can help; indeed, some social and cultural support for expressing positivity is an important tool for maintaining a positive mind-set. However, it may not be the best starting point.

Unfortunately, any public expression becomes a discourse, and any discourse inevitably becomes a dialectic. Any expression of a view in public is seen by some as an invitation to express its opposite⁸. Therefore one either becomes invested in defending the boundaries of a positive community space — a psychically exhausting task in its own right — or one must constantly entertain the possibility that things are, in fact, bad, when one is trying to condition one’s brain to maintain the ability to recognize when things are actually good.

Thus my suggestion to write something for yourself, and only for yourself.

Personally, I use a template that I fill out every day, with four sections:

• “Summary”. Summarize the day in one sentence that encapsulates its positive vibes. Honestly I put this in there because the Notes app (which is what I’m using to maintain this) shows a little summary of the contents of the note, and I was getting annoyed by just seeing “Proud:” as the sole content of that summary. But once I did so, I found that it helps to try to synthesize a positive narrative, as your brain may be constantly trying to assemble a negative one. It can help to write this last, even if it’s up at the top of your note, once you’ve already filled out some of the following sections.

• “I’m proud of:”. First, focus on what you personally have achieved through your skill and hard work. This can be very difficult, if you are someone who has a habit of putting yourself down. Force yourself to acknowledge that you did something useful, even if you didn’t finish anything, you almost certainly made progress and that progress deserves celebration.

• “I’m grateful to:”. Who are you grateful to? Why? What did they do for you? Once you’ve made the habit of allowing yourself to acknowledge your own accomplishments, it’s easy to see those; pay attention to the ways in which others support and help you. Thank them by name.

• “I’m lucky because:”. Particularly in post-2020 hell-world it’s easy to feel like every random happenstance is an aggravating tragedy. But good things happen randomly all the time, and it’s easy to fail to notice them. Take a moment to notice things that went well for no good reason, because you’re definitely going to feel attacked by the universe when bad things happen for no good reason; and they will.

Although such a journal is private, it’s helpful to actually write out the answers, to focus on them, to force yourself to get really specific.

I hope this tool is useful to someone out there. It’s not going to solve any problems, but perhaps it will make the world seem just a little brighter.

As I mentioned previously, I’ve recently been medicated for ADHD.

Everyone’s experience with medication, even the same medication, is different, but my particular experience — while hugely positive — has involved not so much a decrease in symptoms, but rather a shifting of my symptom profile. Some of my executive functions (particularly task initiation) have significantly improved, but other symptoms, such as time blindness have gotten significantly worse. This means, for example, I can now easily decide to perform a task, and actually maintain focus on that task for hours¹, but it’s harder to notice that it’s time to stop, and still somewhat difficult to tear myself away from it.

I’ve tried pomodoro timers before and I’ve had mixed success with them. While I tend to get more done if I set a pomodoro, it’s hard to remember to set the timers in the first place, and it’s hard to do the requisite time-keeping to remember how many pomodoros I’ve already set, how many more I’ll have the opportunity to set, etc. Physical timers have no associated automation and data recording, and apps can be so unobtrusive that I can easily forget about them entirely. I’ve long had an aspiration to eventually write my own custom-tailored app that addresses some of these issues.

As part of a renewed interest in ADHD management techniques, I watched this video about ADHD treatments from Dr. Russell Barkley, wherein he said (I’m paraphrasing) “if I don’t put an intervention into your visual field it might as well not exist”.

I imagined timer that:

1. was always clearly present in my visual field;
2. recorded the passage of intervals of time regardless of any active engagement from the user; the idea is to record the progress of the day, not give you a button you need to remember to push;
3. rewarded me for setting active intentions about what to do with those chunks of time, and allowed me to mark them as successful or failed.

So, last weekend, leveraging my newly enhanced task-initiation and concentration-maintenance abilities, I wrote it, and I’ve been using it all week. Introducing Pomodouroboros, the pomodoro timer that reminds you that the uncaring void marches on regardless of your plans or intentions.

I’ve been using it all week and preliminary results are extremely positive.

This thing is in an extremely rough state. It has no tests, no docs, and an extremely inscrutable UI that you need to memorize in order to use effectively. I need plenty of help with it. I contemplated keeping it private and just shipping a binary, but a realistic assessment of my limited time resources forced me to admit that it already kind of does what I need, and if I want to enhance it to the point where it can help other people, I’ll need plenty of help.

If this idea resonates with you, and you’re on macOS, check out the repo, make a virtualenv somehow, install its dependencies, I don’t know how you make virtualenvs or install dependencies, I’m not your dad², and run ./runme. If you’re on another platform, check out the code, ask me some questions, and maybe try to write a port to one of them.

Some of you read my previous post on typing.Protocols and probably wondered: “what about zope.interface?” I’ve advocated strongly for it in the past — but now that we have Mypy and Protocols, is it simply a relic of an earlier time? Can we entirely replace it with Protocol?

Let’s have a look.

Typing in 2 dimensions

In the previous post I discussed structural versus nominal typing. In Mypy’s type system, most classes are checked nominally whereas Protocol is checked structurally. However, there’s another way that Protocol is distinct from a normal class: normal classes are concrete types, and Protocols are abstract.

Abstract types:

1. cannot be instantiated: every instance of an abstract type is an instance of some concrete sub-type, and
2. do not include (complete) implementation logic.

Concrete types:

1. can be instantiated: they are complete descriptions of a type, and
2. must include all their own implementation logic.

Protocols and Interfaces are both abstract, but Interfaces are nominal. The highest level distinction between the two is that when you have a problem that requires an abstract type, but nominal checking is preferable to structural, Interfaces are a better solution.

Python’s built-in Abstract Base Classes are technically abstract-and-nominal as well, but they’re in a strange halfway space; they’re formally “abstract” because they can’t be instantiated, but they’re partially concrete in that they can contain any amount of implementation logic themselves, and thereby making an object which is a subtype of multiple ABCs drags in all the usual problems of the conflicting namespaces within multiple inheritance.

Theoretically, there’s a way to treat ABCs as purely abstract — which is to use ABCMeta.register — but as of this writing (March 2021) it doesn’t work with Mypy, so within the context of “static typing in Python” we presently have to ignore it.

Practicalities

The first major advantage that Protocol has is that since it is now built in to Python itself, there’s no reason not to use it. When Protocol didn’t even exist, regardless of all the advantages of adding explicit abstract types to your project with zope.interface, it did still have the small down-side of requiring a new dependency, with all the minor headaches that might imply.

beyond the theoretical distinctions, there’s a question of how well tooling supports zope.interface. There are some clear gaps; there is not a ton of great built-in IDE support for zope.interface; less-sophisticated linters will sometimes still complain that Interfaces don’t take self as their first argument. Indeed, Mypy itself does this by default — although more on that in a moment. Less mainstream performance-focused type-checkers like Pyre and Pyright don’t support zope.interface, either, although their lack of support for zope.interface is just a part of a broader problem of their lack of extensibility; they also can’t support SQLAlchemy or the Django ORM without special-casing in the tools themselves.

But what about Mypy itself — if we have to discount ABCMeta.register due to practical tooling deficiencies even if they provide a built-in way to declare a nominal-but-abstract type in principle, we need to be able to use zope.interface within Mypy as well for a fair comparison with Protocol. Can we?

Luckily, yes! Thanks to Shoobx, there’s a fairly actively maintained Mypy plugin that supports zope.interface which you can use to statically check your Interfaces.

However, this plugin does have a few key limitations as of this writing (Again, March 2021), which makes its safety guarantees a bit lower-quality than Protocol.

The net result of this is that Protocols have the “home-field advantage” in most cases; out of the box, they’ll work more smoothly with your existing editor / linter setup, and as long as your project supports Python 3.6+, at worst (if you can’t use Python 3.7, where Protocol is built in to typing) you have to take a type-check-time dependency on the typing_extensions package, whereas with zope.interface you’ll need both the run-time dependency of zope.interface itself and the Mypy plugin at type-checking time.

So in a situation where both are roughly equivalent, Protocol tends to win by default. There are undeniably big areas where Interfaces and Protocols overlap, and in plenty of them, using Protocol is a fine idea. But there are still some clear places that zope.interface shines.

First, let’s look at a case which Interfaces handle more gracefully than Protocols: opting out of matching a simple shape, where the shape doesn’t fully describe its own meaning.

Where Interfaces work best: hidden and complex meanings

The string is a stark data structure and everywhere it is passed there is much duplication of process. It is a perfect vehicle for hiding information.

Alan Perlis, “Epigrams in Programming”, Epigram 34.

The place where structural typing has the biggest advantage is when the type system is expressive enough to fully encode the meaning of the desired behavior within the structure of the type itself. Consider a Protocol which describes an object that can add some integers together:

1
2
3

class Math(Protocol):
    def add_integers(addend1: int, addend2: int) -> int:
        ...

It’s fairly unambiguous what adherents to this Protocol should do, and anyone implementing such a thing should be able to clearly tell that the method is supposed to add a couple of integers together; there’s nothing hidden about the structure of the integers, no constraints the type system won’t let us specify. It would be quite surprising if anything that didn’t have the intended behavior would match this Protocol.

A the other end of the spectrum, we might have a plugin Interface that has a lot of hidden structure. For this example, we have an Interface called IPlugin containing a method with an easy-to-conflict-with name (“name”) overloaded with very specific constraints on its return type: the string must contain the dotted-path name of a Python object in an import-able module (like, for example, "os.path.join").

1
2
3

class IPlugin(Interface):
    def name() -> str:
        "Return the fully-qualified Python identifier of the thing to load."

With Protocols, you can work around these limitations, by manually making it harder to match; adding elements to the structure that embed names relevant to its semantics and thereby making the type behave more as if it were nominally typed.

You could make the method’s name long and ugly instead (plugin_name_to_load, let’s say) or add unused additional attributes (yep_i_am_a_plugin = Literal[True]) in order to reduce the risk of accidental matches, but these workarounds look hacky, and they have to be manually namespaced; if you want to mark it as having semantics associated with your specific plugin system, you have to embed the name of that system in your attributes themselves; here we’re just saying “plugin” but if we want to be truly careful, we have to embed the whole name of our project in there.

With Interfaces, the maintainer of each implementation must explicitly opt in, by choosing whether to specify that they are an @implementer(IPlugin). Since they had to import IPlugin from somewhere, this annotation carries with it a specific, namespaced declaration of semantic intent: “I know what the Interface IPlugin means, and I promise that I can provide it”.

This is the most salient distinction between Protocols and Interfaces: if you have strong reasons to want adherents to the abstract type to opt in, you want an Interface; if you want them to match automatically, you want a Protocol.

Runtime support

Interfaces also provide a more nuanced set of runtime checks.

You can say that an object directlyProvides an interface, allowing for some level of (at least runtime) type safety, and ask if IPlugin is .providedBy some object.

You can do most of this with Protocol, but it’s awkward. The @runtime_checkable decorator allows your Protocol to make isinstance(x, MyProtocol) work like IMyInterface.providedBy(x), but:

1. you’re still missing directlyProvides; the runtime checking is all by type, not by the individual properties of the instance;
2. it’s not the default, so if you’re not the one defining the Protocol, there’s no guarantee you’ll be able to use it.

With Interfaces, there’s also no mandatory relationship between the implementer (i.e. the type whose instances fit the specified shape) and the provider (the specific object which can fit the specified shape). This means you get features like classProvides and moduleProvides “for free”.

Interfaces work particularly well for communication between frameworks and application code. For example, let’s say you’re evolving the meaning of an Interface implemented by applications over time — EventHandler, EventHandler2, EventHandler3 — which have similarly named and typed methods, but subtly different expectations on their lifecycle or when precisely the methods will be called. A framework facing this problem can use a series of Interfaces, and check at runtime to see which of these the application implements, and be secure in the knowledge that the application has properly intentionally adopted the new interface, and doesn’t just happen to have a matching method name against an older version.

Finally, zope.interface gives you adaptation and adapter registries, which can be a useful mechanism for doing things like templating, like a much more powerful version of singledispatch from the standard library.

Adapter registries are nuanced, complex tools and unfortunately an example that captures the full utility of their power would itself be commensurately complex. However, the core of adaptation is the idea that if you have an arbitrary object x, and you want a provider of the interface IY, you can do the following:

1

y = IY(x, None)

This performs a multi-stage check:

1. If x already provides IY (either via implementer, provider, directlyProvides, classProvides, or moduleProvides), it’s simply returned; so you don’t need to special-case the case where you’ve already got what you want.
2. If x has a __conform__(interface) method, it’ll be called with IY as the interface, and if __conform__ returns anything non-None that result will be returned from the call to IY.
3. If IY has a specially-defined __adapt__ method, it can implement its own logic for this hook directly.
4. Each globally-registered function in zope.interface’s adapter_hooks will be invoked to find a function that can transform x into an IY provider. Twisted has its own global registry in this list, which is what registerAdapter manipulates.

But from the perspective of the caller, you can just say “I want an IY”.

With Protocols, you can emulate this with functools.singledispatch by making a function which returns your Protocol type and registers various types to do conversion. The place that adapter registries have an advantage is their central nature and consistent idiom for converting to the target type; you can use adaptation for any Interface in the same way, and any type can participate in adaptation in the ways listed above via flexible mechanisms depending on where it makes sense to put your implementation, whereas any singledispatch function to convert to a Protocol needs to be bespoke per-Protocol.

Describing and restricting existing shapes

There are still several scenarios where Protocol’s semantics apply more cleanly.

Unlike Interfaces, Protocols can describe the types of things that already exist. To see when that’s an advantage, consider a sprawling application that uses tons of libraries and manipulates 3D spatial data points.

There’s a convention among these disparate libraries where they all represent a “point” as an object with .x, .y, and .z attributes which are all floats. This is a natural enough shape, given the domain, that lots of your libraries just fit it by accident. You want to write functions that can work with data output by any of these libraries as long as it plausibly looks like your own concept of a Point:

1
2
3
4

class Point(Protocol):
    x: float
    y: float
    z: float

In this case, the thing defining the Protocol is your application; the thing implementing the Protocol is your collection of libraries. Since the libraries don’t and can’t know about the application — the dependency arrow points the other way — they can’t reference the Protocol to note that they implement it.

Using Protocol, you can also restrict an existing type to preserve future flexibility.

For example, let’s say we’re implementing a “mailbox” type pattern, where some systems deliver messages and other systems retrieve them later. To avoid mix-ups, the system that sends the messages shouldn’t retrieve them and vice versa - receivers only receive, and senders only send. With Protocols, we can describe this without having any new custom concrete types, like so:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

from typing import Protocol, TypeVar

T_co = TypeVar("T_co", covariant=True)
T_con = TypeVar("T_con", contravariant=True)

class Sender(Protocol[T_con]):
    def add(self, item: T_con) -> None:
        "Put an item in the slot."

class Receiver(Protocol[T_co]):
    def pop(self) -> T_co:
        "Retrieve an item from the PO box."

All of that code is just telling Mypy our intentions; there’s no behavior here yet.

The actual implementation is even shorter:

1
2
3

from typing import Set

mailbox: Set[int] = set()

Literally no code of our own - set already does the job we described. And how do we use this?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

def send(sender: Sender[int]) -> None:
    sender.add(3)

def receive(receiver: Receiver[int]) -> None:
    receiver.pop()
    receiver.add(3)
    # Mypy stops us making this mistake:
    # "Receiver[int]" has no attribute "add"

send(mailbox)
receive(mailbox)

For its initial implementation, this system requires nothing beyond types available in the standard library; just a set. However, by treating their parameter as a Sender and a Receiver respectively rather than a Set, send and receive prevent themselves from using any functionality from the set passed in aside from the one method that their respective roles are supposed to “see”. As a result, Mypy will now tell us if any code which receives the sender object tries to remove objects.

This allows us to use existing data structures in libraries without the usual attendant problem of advertising to all clients that every tiny implementation detail of those existing structures is an intended part of the public interface. Python has always tried to make these sort of distinctions by leaving certain things undocumented or saying narratively which things you should rely on, but it’s always hit-or-miss (usually miss) whether library consumers will see those admonitions or not; by making it a feature of the programming environment, Mypy makes it harder to ignore.

Conclusions

In modern Python code, when you have an abstract collection of behavior, you should probably consider using a Protocol to describe it by default. However, Interface is also staying up to date with modern Python tooling by with Mypy support, and it can be worthwhile for more sophisticated consumers that want support for nominal typing, or that want to draw on its reach adaptation and component registration feature-set.

One of my favorite features within Twisted — but also one of the least known — is LoopingCall.withCount, which can be used in applications where you have some real-time thing happening, which needs to keep happening at a smooth rate regardless of any concurrent activity or pauses in the main loop. Originally designed for playing audio samples from a softphone without introducing a desync delay over time, it can also be used to play animations while keeping track of their appropriate frame.

LoopingCall is all around a fun tool to build little game features with. I’ve built a quick little demo to showcase some discoveries I’ve made over a few years of small hobby projects (none of which are ready for an open-source release) over here: DrawSnek.

This little demo responds to 3 key-presses:

1. q quits. Always a useful thing for full-screen apps which don’t always play nice with C-c :).
2. s spawns an additional snek. Have fun, make many sneks.
3. h introduces a random “hiccup” of up to 1 full second so you can see what happens visually when the loop is overburdened or stuck.

Unfortunately a fully-functioning demo is a bit lengthy to go over line by line in a blog post, so I’ll just focus on a couple of important features for stutter- and tearing-resistant animation & drawing with PyGame & Twisted.

For starters, you’ll want to use a very recent prerelease of PyGame 2, which recently added support for vertical sync even without OpenGL mode; then, pass the vsync=1 argument to set_mode:

1
2
3
4
5

screen = pygame.display.set_mode(
    (640 * 2, 480 * 2),
    pygame.locals.SCALED | pygame.locals.FULLSCREEN,
    vsync=1
)

To allow for as much wall-clock time as possible to handle non-drawing work, such as AI and input handling, I also use this trick:

1
2
3
4
5
6
7

def drawScene():
    screen.fill((0, 0, 0))
    for drawable in self.drawables:
        drawable.draw(screen)
    return deferToThread(pygame.display.flip)

LoopingCall(drawScene).start(1 / 62.0)

By deferring pygame.display.flip to a thread¹, the main loop can continue processing AI timers, animation, network input, and user input while blocking and waiting for the vertical blank. Since the time-to-vblank can easily be up to 1/120th of a second, this is a significant amount of time! We know that the draw won’t overlap with flip, because LoopingCall respects Deferreds returned from its callable and won’t re-invoke you until the Deferred fires.

Drawing doesn’t use withCount, because it just needs to repeat about once every refresh interval (on most displays, about 1/60th of a second); the vblank timing is what makes sure it lines up.

However, animation looks like this:

1
2
3

def animate(self, frameCount):
    self.index += frameCount
    self.index %= len(self.images)

We move the index forward by however many frames it’s been, then be sure it wraps around by modding it by the number of frames.

Similarly, the core² of movement looks like this:

1
2
3

def move(self, frameCount):
    self.sprite.x += frameCount * self.dx
    self.sprite.y += frameCount * self.dy

Rather than moving based on the number of times we’ve been called, which can result in slowed-down movement when the framerate isn’t keeping up, we jump forward by however many frames we should have been called at this point in time.

One of these days, maybe I’ll make an actual game, but in the meanwhile I hope you all enjoy playing with these fun little basic techniques for using Twisted in your game engine.