Dates And Times And Types

Get a TypeError when using a datetime when you wanted a date.

Python’s standard datetime module is very powerful. However, it has a couple of annoying flaws.

Firstly, datetimes are considered a kind of date1, which causes problems. Although datetime is a literal subclass of date so Mypy and isinstance believe a datetime “is” a date, you cannot substitute a datetime for a date in a program without provoking errors at runtime.

To put it more precisely, here are two programs which define a function with type annotations, that mypy finds no issues with. The first of which even takes care to type-check its arguments at run-time. But both raise TypeErrors at runtime:

Comparing datetime to date:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
from datetime import date, datetime

def is_after(before: date, after: date) -> bool | None:
    if not isinstance(before, date):
        raise TypeError(f"{before} isn't a date")
    if not isinstance(after, date):
        raise TypeError(f"{after} isn't a date")
    if before == after:
        return None
    if before > after:
        return False
    return True

is_after(date.today(), datetime.now())
1
2
3
4
5
6
Traceback (most recent call last):
  File ".../date_datetime_compare.py", line 14, in <module>
    is_after(date.today(), datetime.now())
  File ".../date_datetime_compare.py", line 10, in is_after
    if before > after:
TypeError: can't compare datetime.datetime to datetime.date

Comparing “naive” and “aware” datetime:

1
2
3
4
5
6
from datetime import datetime, timezone, timedelta

def compare(a: datetime, b: datetime) -> timedelta:
    return a - b

compare(datetime.now(), datetime.now(timezone.utc))
1
2
3
4
5
6
Traceback (most recent call last):
  File ".../naive_aware_compare.py", line 6, in <module>
    compare(datetime.now(), datetime.now(timezone.utc))
  File ".../naive_aware_compare.py", line 4, in compare
    return a - b
TypeError: can't subtract offset-naive and offset-aware datetimes

In some sense, the whole point of using Mypy - or, indeed, of runtime isinstance checks - is to avoid TypeError getting raised. You specify all the types, the type-checker yells at you, you fix it, and then you can know your code is not going to blow up in unexpected ways.

Of course, it’s still possible to avoid these TypeErrors with runtime checks, but it’s tedious and annoying to need to put a check for .tzinfo is not None or not isinstance(..., datetime) before every use of - or >.

The problem here is that datetime is trying to represent too many things with too few types. datetime should not be inheriting from date, because it isn’t a date, which is why > raises an exception when you compare the two.

Naive datetimes represent an abstract representation of a hypothetical civil time which are not necessarily tethered to specific moments in physical time. You can’t know exactly what time “today at 2:30 AM” is, unless you know where on earth you are and what the rules are for daylight savings time in that place. However, you can still talk about “2:30 AM” without reference to a time zone, and you can even say that “3:30 AM” is “60 minutes after” that time, even if, given potential changes to wall clock time, that may not be strictly true in one specific place during a DST transition. Indeed, one of those times may refer to multiple points in civil time at a particular location, when attached to different sides of a DST boundary.

By contrast, Aware datetimes represent actual moments in time, as they combine civil time with a timezone that has a defined UTC offset to interpret them in.

These are very similar types of objects, but they are not in fact the same, given that all of their operators have slightly different (albeit closely related) semantics.

Using datetype

I created a small library, datetype, which is (almost) entirely type-time behavior. At runtime, despite appearances, there are no instances of new types, not even wrappers. Concretely, everything is a date, time, or datetime from the standard library. However, when type-checking with Mypy, you will now get errors reported from the above scenarios if you use the types from datetype.

Consider this example, quite similar to our first problematic example:

Comparing AwareDateTime or NaiveDateTime to date:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
from datetype import Date, NaiveDateTime

def is_after(before: Date, after: Date) -> bool | None:
    if before == after:
        return None
    if before > after:
        return False
    return True

is_after(Date.today(), NaiveDateTime.now())

Now, instead of type-checking cleanly, it produces this error, letting you know that this call to is_after will give you a TypeError.

1
2
date_datetime_datetype.py:10: error: Argument 2 to "is_after" has incompatible type "NaiveDateTime"; expected "Date"
Found 1 error in 1 file (checked 1 source file)

Similarly, attempting to compare naive and aware objects results in errors now. We can even use the included AnyDateTime type variable to include a bound similar to AnyStr from the standard library to make functions that can take either aware or naive datetimes, as long as you don’t mix them up:

Comparing AwareDateTime to NaiveDateTime:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
from datetime import datetime, timezone, timedelta
from datetype import AwareDateTime, NaiveDateTime, AnyDateTime


def compare_same(a: AnyDateTime, b: AnyDateTime) -> timedelta:
    return a - b


def compare_either(
    a: AwareDateTime | NaiveDateTime,
    b: AwareDateTime | NaiveDateTime,
) -> timedelta:
    return a - b


compare_same(NaiveDateTime.now(), AwareDateTime.now(timezone.utc))

compare_same(AwareDateTime.now(timezone.utc), AwareDateTime.now(timezone.utc))
compare_same(NaiveDateTime.now(), NaiveDateTime.now())
1
2
3
4
5
6
naive_aware_datetype.py:13: error: No overload variant of "__sub__" of "_GenericDateTime" matches argument type "NaiveDateTime"
...
naive_aware_datetype.py:13: error: No overload variant of "__sub__" of "_GenericDateTime" matches argument type "AwareDateTime"
...
naive_aware_datetype.py:16: error: Value of type variable "AnyDateTime" of "compare_same" cannot be "_GenericDateTime[Optional[tzinfo]]"
Found 3 errors in 1 file (checked 1 source file)

Telling the Difference

Although the types in datetype are Protocols, there’s a bit of included magic so that you can use them as type guards with isinstance like regular types. For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
from datetype import NaiveDateTime, AwareDateTime
from datetime import datetime, timezone

nnow = NaiveDateTime.now()
anow = AwareDateTime.now(timezone.utc)


def check(d: AwareDateTime | NaiveDateTime) -> None:
    if isinstance(d, NaiveDateTime):
        print("Naive!", d - nnow)
    elif isinstance(d, AwareDateTime):
        print("Aware!", d - anow)


check(NaiveDateTime.now())
check(AwareDateTime.now(timezone.utc))

Try it out, carefully

This library is very much alpha-quality; in the process of writing this blog post, I made a half a dozen backwards-incompatible changes, and there are still probably a few more left as I get feedback. But if this is a problem you’ve had within your own codebases - ensuring that dates and datetimes don’t get mixed up, or requiring that all datetimes crossing some API boundary are definitely aware and not naive, give it a try with pip install datetype and let me know if it catches any bugs!


  1. But, in typical fashion, not a kind of time... 

Leave The Frog For Last

Neurotypical advice for ADHD is not always great.

This was originally a thread on Twitter; you can read the original here, but this one has been lightly edited for grammar and clarity, plus I added a pretty rad picture of a frog to it.

Update 2022-05-16: Thanks to some reader feedback I have updated the conclusion to note an example where this advice can productively apply to some ADHDers.

I’m in the midst of trying to unlearn a few things about neurotypical productivity advice but this is one I’ve been thinking about a lot:

“Eat the frog first” is particularly toxic advice for ADHDers.

A frog on a flower, nervously looking at you as you contemplate whether to eat it.

Photo by Stephanie LeBlanc on Unsplash

First, for anyone who happens not to know already: “eat the frog first” is a technique which involves finding the task you’re most likely to ignore or put off, and doing it first in your day to ensure that you don’t avoid it.

For a neurotypical person, eating the frog first makes sense, which is of course why this advice exists in the first place. If you’ve been avoiding a task, put it first in your day when you’re going to have the most energy, and use the allure of the more fun tasks later to push through it.

This makes intuitive sense.

The premise of this advice is that you rely on the promise of delayed gratification—and the anticipated inherent satisfaction of having completed the boring and unpleasant thing—in order to motivate you to do it.

Here’s the problem for ADHDers: ADHD is literally the condition of not generating enough dopamine, which means delayed gratification is inherently more difficult for us. The anticipated inherent satisfaction is less motivating because it’s less intense, on a physical level.

An ADHD brain powering through tasks needs momentum. You need to be in a sufficiently excited state to begin doing things. A bored, dopamine-starved ADHD brain is going to be clawing at the walls looking for ANY dopamine-generating distraction to avoid thinking about the frog.

Of course where dopamine won’t do, there’s always adrenaline. Panic can trigger sufficient states of activity as well, although the stress is unhealthy and it’s less reliable in the absence of a real, immediate threats that you can’t ignore.

So what frog-first ADHD days often look like (particularly for adult ADHDers) is a slow slog of not really doing anything useful, while stewing in increasingly negative self-talk, attempting to generate the necessary anger and self-loathing required to truly panic about the frog.

Unfortunately this type of attempt at internal motivation is more likely to result in depression than motivation, which creates a spiral that makes the problem worse.

The neurotypical’s metaphorical frog is just sitting there, waiting to be eaten. Maybe they’ve been avoiding it because it’s a little gross, but fine, they can exert a little willpower and just do it, and move on to more pleasant activities. But the ADHD frog is running away.

Trying to use the same technique just results in the ADHDer sitting in the swamp where the frog used to be, chugging ever-increasing volumes of toxic mud in the hopes that we’ll find a frog in there. Sometimes we even find one! But that’s not success.

At the end of the day, the metaphorical frog does need eating; that’s what makes it a frog. What is the conscientious ADHDer to do?

Unfortunately, there is no singular, snappy answer; difficulty with this type of task is the impenetrable core of the “disorder” part of ADHD. It’ll always be difficult. But there are definitely strategies which can make it relatively easier.

None of these are guaranteed to work, but I am at least reasonably sure that they won’t build a spiral into guilt and depression:

  1. start with a fun task, and build momentum until the frog seems like no big deal
  2. use hype music; yell; get excited to an embarrassing degree.1
  3. exercise; i.e. “go for a walk”

It might literally be better to start the day with something actively unproductive, but fun, like a video game, although this can obviously be risky. For this to work, you need to have very good systems in place.

Start the frog at the end of the day and deliberately interrupt yourself when you stop work. Leave it lingering so some aspect of it annoys you and it distracts you at night. Start the next day pissed off at and obsessing over murdering that piece of shit frog as soon as you can get your hands on it.

This technique is also good because at the end of the day you only need to push yourself just hard enough to load the task into your brain, not all the way through it.

Remember that while “stimulated” doesn’t have to mean “panicked”, it also doesn’t need to mean “happy”. Sometimes, annoyance or irritation is the best way to ensure that you go do something. Consider, for example, the compelling motivation of reading a comment on the Internet that you disagree with.

Overall the distinguishing characteristic of toxic productivity advice is that it makes you spend more time feeling bad than doing stuff. It substitutes panic for healthy motivation, and low self-esteem for a feeling of accomplishment.

The most important point I am trying to make is this: when you take productivity advice — even, or perhaps especially, from me – try to measure its impact on your work and your mental health.

To that point, one piece of feedback I received on an earlier iteration of this article was that, for some ADHDers on stimulant medication2, eating the frog first can work: if you take your medication early in the morning and experience a big, but temporary, increase to executive-function 30 minutes later, being prepared to do your frog-eating at that specific moment can have similar results as for someone more neurotypical. This very much depends on how you specifically react to your medication, however.

So, if eating the frog first is working for you, by all means keep doing it, but you have to ask yourself: are you actually getting more done?


  1. One of the advantages of working from home is that you can really lean into this without provoking an intervention from your coworkers. 

  2. I personally take a slightly unusual kind of ADHD medication, which does help but not in the typical fashion. 

Inbox Zero, Cost: Zero

Updated guidance for getting out of an email overwhelm trap, with practical, concrete, free examples.

One consistent bit of feedback that I’ve received on my earlier writing about email workflow is that I didn’t include a concrete enough set of instructions for getting started with task-management workflow, particularly with low-friction options that are available for people who don’t necessarily have $100 per year to drop on the cadillac of task-management applications.

Given that the piece seems to be enjoying a small resurgence of attention, I’ve significantly expanded the “Make A Place For Tasks” section of that article, with:

  • more no-cost, low-friction options for getting started (if you’re stuck on this step “if you use Gmail, just start using Google Tasks” is the main takeaway)
  • a guide for how to evaluate a task-management application for yourself, if you are trying to pick something that fits your work style better
  • several links to the specific “create a task from an email” tools and workflows for each app

It was nice to be doing this update now, because in the years since that piece was published, almost every major email application has added task-management features, or upgraded them into practical usability; gone are the times when properly filing your emails into clearly-described tasks was an esoteric feature that you needed expensive custom software for.

You Should Compile Your Python And Here’s Why

write Python that’s faster than C by optimizing your code, adding standard type annotations, and using Mypyc.

In this post I’d like to convince you that you should be running Mypyc over your code1 — especially if your code is a library you upload to PyPI — for both your own benefit and that of the Python ecosystem at large.

But first, let me give you some background.

Python is Slow, And That’s Fine, Because It’s Fast Enough

A common narrative about Python’s value proposition, from the very earliest days of the language2, often recited in response to a teammate saying “shouldn’t we just write this in $HIGHER_PERFORMANCE_LANGUAGE instead?” goes something like this:

Sure, Python is slow.

But that’s okay, because it saves you so much time over implementing your code in $HIGHER_PERFORMANCE_LANGUAGE that you’ll have so much more time for optimizing those critical hot-spots where performance is really critical.

And if the language’s primitives are too slow to micro-optimize those hot-spots enough, that’s okay too, because you can always re-write just those small portions of the program as a C extension module.

Python’s got you covered!

There is some truth to this narrative, and I’ve quoted from it myself on many occasions. When I did so, I was not quoting it as some facile, abstract hypothetical, either. I had a few projects, particularly very early in my Python career, where I replaced performance-critical C++ code with a one tenth the number of lines of Python, and improved performance by orders of magnitude in the process3.

When you have algorithmically interesting, performance-sensitive code that can benefit from a high-level expressive language, and the resources to invest in making it fast, this process can be counterintuitively more efficient than other, “faster” tools. If you’re working on massively multiplayer online games4 or something equally technically challenging, Python can be a surprisingly good idea.

But… Is It Fine, Though?

This little nugget of folk wisdom does sound a bit defensive, doesn’t it? If Python were just fast, you could just use it, you wouldn’t need this litany of rationalizations. Surely if we believed that performance is important in our own Python code, we wouldn’t try to wave away the performance of Python itself.

Most projects are not massively multiplayer online games. On many straightforward business automation projects, this sort of staged approach to performance is impractical.

Not all performance problems are hot spots. Some programs have to be fast all the way through. This is true of some complex problems, like compilers and type checkers, but is also often the case in many kinds of batch processing; there are just a lot of numbers, and you have to add them all up.

More saliently for the vast majority of average software projects, optimization just isn’t in the budget. You do your best on your first try and hope that none of those hot spots get too hot, because as long as the system works within a painfully generous time budget, the business doesn’t care if it’s slow.

The progression from “idiomatic Python” to “optimized Python” to “C” is a one-way process that gradually loses the advantages that brought us to Python in the first place.

The difficult-to-reverse nature of each step means that once you have prototyped out a reasonably optimized data structure or algorithm, you need to quasi-permanently commit to it in order to squeeze out more straight-line performance of the implementation.

Plus, the process of optimizing Python often destroys its readability, for a few reasons:

  1. Optimized Python relies on knowledge of unusual tricks. Things like “use the array module instead of lists”, and “use % instead of .format”.
  2. Optimized Python requires you to avoid the things that make Python code nicely organized:
    1. method lookups are slow so you should use functions.
    2. object attribute accesses are slow so you should use tuples with hard-coded numeric offsets.
    3. function calls are slow so you should copy/paste and inline your logic
  3. Optimized Python requires very specific knowledge of where it’s going to be running, so you lose the flexibility of how to run it: making your code fast on CPython might make it much slower on PyPy, for example. Native extension modules can make your code faster, but might also make it fail to run inside a browser, or add a ton of work to get it set up on a new operating system.

Maintaining good performance is part of your software’s development lifecycle, not just a thing you do once and stop. So by moving into this increasingly arcane dialect of “fast” python, and then into another programming language entirely with a C rewrite, you end up having to maintain C code anyway. Not to mention the fact that rewriting large amounts of code in C is both ludicrously difficult (particularly if your team primarily knows Python) and also catastrophically dangerous. In recent years, safer tools such as PyO3 have become available, but they still involve switching programming languages and rewriting all your code as soon as you care about speed5.

So, for Python to be a truly general-purpose language, we need some way to just write Python, and have it be fast.

It would benefit every user of Python for there to be an easy, widely-used way to make idiomatic, simple Python that just does stuff like adding numbers, calling methods, and formatting strings in a straight line go really fast — exactly the sorts of things that are the slowest in Python, but are also the most common, particularly before you’ve had an opportunity to cleverly optimize.

We’ve Been Able To At Least Make Do

There are also a number of tools that have long been in use for addressing this problem: PyPy, Pyrex, Cython, Numba, and Numpy to name a few. Their maintainers all deserve tremendous amounts of credit, and I want to be very clear that this post is not intended to be critical of anyone’s work here. These tools have drawbacks, but many of those drawbacks make them much better suited to specialized uses beyond the more general 80% case I’m talking about in this post, for which Mypyc would not be suitable.

Each one of these tools impose limitations on either the way that you write code or where you can deploy it.

Cython and Numba aren’t really “Python” any more, because they require special-purpose performance-oriented annotations. Cython has long supported pure-Python type annotations, but you won’t get any benefit from telling it that your variable is an int, only a cython.int. It can’t optimize a @dataclass, only a @cython.cclass. And so on.

PyPy gets the closest — it’s definitely regular Python — but its strategy has important limitations. Primarily, despite the phenomenal and heroic effort that went into cpyext, it seems like there’s always just one PyPy-incompatible library in every large, existing project’s dependency list which makes it impossible to just drop in PyPy without doing a bunch of arcane debugging first.

PyPy might make your program magically much faster, but if it doesn’t work, you have to read the tea leaves on the JIT’s behavior in a profiler which practically requires an online component that doesn’t even work any more. So mostly you just simplify your code to use more straightforward data structures and remove CPython-specific tricks that might trip up the JIT, and hope for the best.

PyPy also introduces platform limitations. It’s always — understandably, since they have to catch up after the fact — lagging a bit behind the most recently released version of CPython, so there’s always some nifty language feature that you have to refrain from using for at least one more release cycle.

It also has architectural limitations. For example, it performs quite poorly on an M1 Mac since it still runs under x86_64 emulation on that platform. And due to iOS forbidding 3rd-party JITs, it won’t ever be able to provide better performance in one of the more constrained environments that needs it more that other places. So you might need to rely on CPython on those platforms anyway… and you just removed all your CPython-specific hacks to try to please the JIT on the other platforms you support.

So while I would encourage everyone to at least try their code on PyPy — if you’re running a web-based backend, it might save you half your hardware budget6 — it’s not going to solve “python is slow” in the general case.

It’ll Eventually Be All Right

This all sounds pretty negative, so I would be remiss if I did not also point out that the core team is well aware that Python’s default performance needs to be better, and Guido van Rossum literally came out of retirement for one last job to fix it, and we’ve already seen a bunch of benefits from that effort.

But there are some fundamental limitations on the long-term strategy for these optimizations; one of the big upcoming improvements is a JIT, which suffers from some (but not all) of the same limitations as PyPy, and the late-bound, freewheeling nature of Python inherently comes with some performance tradeoffs.

So it would still behoove us to have a strategy for production-ized code that gives good, portable, ahead-of-time performance.

But What About Right Now?

Mypyc takes the annotations meant for Mypy and generates C with them, potentially turning your code into a much more efficient extension module. As part of Mypy itself, it does this with your existing Python type-hints, the kind you’d already use Mypy with to check for correctness, so it doesn’t entail much in the way of additional work.

I’d been curious about this since it was initially released, but I still haven’t had a hard real-world performance problem to really put it through its paces.

So when I learned about the High Throughput Fizzbuzz Challenge via its impressive assembler implementation that achieves 56GiB/s, and I saw even heavily-optimized Python implementations sitting well below the performance of a totally naïve C reference implementation, I thought this would be an interesting miniature experiment to use to at least approximate practical usage.

In Which I Design A Completely Unfair Fight Which I Will Then Handily Win

The dizzying heights of cycle-counting hand-tuned assembler implementations of this benchmark are squarely out of our reach, but I wanted to see if I could beat the performance of this very naïve C implementation with Python that was optimized, but at least, somewhat idiomatic and readable.

I am about to compare a totally naïve C implementation with a fairly optimized hand-tuned Python one, which might seem like an unfair fight. But what I’m trying to approximate here is a micro-instance of the real-world development-team choice that looks like this:

Since Python is more productive, but slower, the effort to deliver each of the following is similar:

  1. a basic, straightforward implementation of our solution in C
  2. a moderately optimized Python implementation of our solution

and we need to choose between them.

This is why I’ll just be showing naïve C and not unrolling any loops; I’ll use -O3 because any team moderately concerned with performance would at least turn on the most basic options, but nothing further.

Furthermore, our hypothetical team also has this constraint, which really every reasonable team should:

We can trade off some readability for efficiency, but it’s important that our team be able to maintain this code going forward.

This is why I’m doing a bit of optimizing in Python but not going all out by calling mmap or pulling in numpy or attempting to use something super esoteric like a SIMD library to emulate what the assembler implementations do. The goal is that this is normal Python code with a reasonable level of systems-level understanding (i.e. accounting for the fact that pipes have buffers in the kernel and approximately matching their size maximizes throughput).

If you want to see FizzBuzz pushed to its limit, you can go check out the challenge itself. Although I think I do coincidentally beat the performance of the Python versions they currently have on there, that’s not what I’m setting out to do.

So with that elaborate framing of this slightly odd experiment out of the way, here’s our naïve C version:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
#include <stdio.h>

int main() {
    for (int i = 1; i < 1000000000; i++) {
        if ((i % 3 == 0) && (i % 5 == 0)) {
            printf("FizzBuzz\n");
        } else if (i % 3 == 0) {
            printf("Fizz\n");
        } else if (i % 5 == 0) {
            printf("Buzz\n");
        } else {
            printf("%d\n", i);
        }
    }
}

First, let’s do a quick head-to-head comparison with a naïve Python implementation of the algorithm:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
def fizzbuzz() -> None:
    for counter in range(1, 1000000000):
        fizz = counter % 3 == 0
        buzz = counter % 5 == 0
        if fizz:
            print("Fizz", end="")
        if buzz:
            print("Buzz", end="")
        if not (fizz or buzz):
            print(counter, end="")
        print()

if __name__ == "__main__":
    fizzbuzz()

Running both of these on my M1 Max MacBook, the naïve C implementation yields 127 MiB/s of Fizzbuzz output. But, as I said, although we’re not going to have time for testing a more complex optimized C version, we would want to at least build it with the performance benefits we get for free with the -O3 compiler option. It turns out that yields us a 27 MiB/s speedup. So 154 MiB/s is the number we have to beat.7

The naïve Python version achieves a dismal 24.3 MiB/s, due to a few issues. First of all, although it’s idiomatic, print() is doing a lot of unnecessary work here. Among other things, we are encoding Unicode, which the C version isn’t. Still, our equivalent of adding the -O3 option for C is running mypyc without changing anything, and that yields us a 6.8MiB/s speedup immediately. We still aren’t achieving comparable performance, but a roughly 25% performance improvement for no work at all is a promising start!8

In keeping with the “some optimizations, but not so much that it’s illegible” constraint described above, the specific optimizations I’ve chosen to pursue here are:

  1. switch to using bytes objects and sys.stdout.buffer to avoid encoding overhead
  2. take advantage of the repeating nature of the pattern in FizzBuzz output and pre-generate a template rather than computing each line independently
  3. fill out the buffer with the relevant integers from a sequence as we go
  4. tune the repetition of that template to a size that roughly fills a pipe buffer on my platform of choice

Hopefully, with that explanation, this isn’t too bad:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
from sys import stdout
from typing import Tuple, Iterable


def precompute_template() -> Iterable[bytes]:
    for counter in range(1, 16):
        fizz = counter % 3 == 0
        buzz = counter % 5 == 0
        if fizz:
            yield b"Fizz"
        if buzz:
            yield b"Buzz"
        if not (fizz or buzz):
            yield b"%d"
        yield b"\n"


chunk_copies = 4
precomputed_template_chunks = list(precompute_template())
format_string = b"".join(precomputed_template_chunks)
number_indexes = [
    number_index
    for number_index, line_content in enumerate(format_string.split(b"\n"))
    if line_content == b"%d"
]
format_string *= chunk_copies


def fizzbuzz() -> None:
    num: int = 1
    output = stdout.buffer.write
    for num in range(1, 1000000001, 15 * chunk_copies):
        t: Tuple[int, ...] = tuple(
            (
                x + number_index
                for x in range(num, num + (15 * chunk_copies), 15)
                for number_index in number_indexes
            )
        )
        output(format_string % t)


if __name__ == "__main__":
    fizzbuzz()

Running this optimized version actually gets us within the ballpark of the naïve C version, even beating it by a hair; my measurement was 159 MiB/s, a small improvement even over -O3. So, per the “litany against C” from the beginning of this post, algorithmic optimization of Python really does help a lot; it’s not just a rationalization. This is a much bigger boost than our original no-effort Mypyc run, giving us more like an 85% speedup; definitely bigger than 25%.

But clearly we’re still being slowed down by Python’s function call overhead, object allocations for small integers, and so on, so Mypyc should help us out here: and indeed it does. On my machine, it nets a whopping 233 MiB/s. Now that we are accounting for performance and optimizing a bit, Mypyc’s relative advantage has doubled to a 50% improvement in performance on both the optimized-but-interpreted Python and naïve C versions.

It’s worth noting that the technique I used to produce the extension modules to test was literally pip install mypy; mypyc .../module.py, then python -c “import module”. I did already have a C compiler installed, but other than that, there was no setup.

I just wrote Python, and it just worked.

The Call To Adventure

Here’s what I want you to take away from all this:

  1. Python can be fast.
  2. More importantly, your Python can be fast.
  3. For a fairly small investment of effort, your Python code can be made meaningfully faster.

Unfortunately, due to the limitations and caveats of existing powerful performance tools like Cython and PyPy, over the last few years in the Python community a passive consensus has emerged. For most projects, in most cases, it’s just not worth it to bother to focus on performance. Everyone just uses the standard interpreter, and only fixes the worst performance regressions.

We should, of course, be glad that the standard interpreter is reliably getting faster all the time now, but we shouldn’t be basing our individual libraries’ and applications’ performance strategies on that alone.

The projects that care the most about performance have made the effort to use some of these tools, and they have often invested huge amounts of effort to good effect, but often they care about performance too much. They make the problem look even harder for everyone else, by essentially stipulating that step 1 is to do something extreme like give up and use Fortran for all the interesting stuff.

My goal with this post is to challenge that status quo, spark interest in revisiting the package ecosystem’s baseline performance expectations, and to get more projects — particularly libraries on PyPI — to pick up Mypyc and start giving Python a deserved reputation for being surprisingly fast.

The Last Piece of the Puzzle

One immediate objection you might be thinking of is the fact that, under the hood, Mypyc is emitting some C code and building it, and so this might create a problem for deployment: if you’ve got a Linux machine but 30% of your users are on Windows, moving from pure-Python to this hybrid workflow might create installation difficulties for them, or at least they won’t see the benefits.

Luckily a separate tool should make that a non-issue: cibuildwheel. “CI Build Wheel”, as its name suggests, lets you build your wheels in your continuous integration system, and upload those builds automatically upon tagging a release.

Often, the bulk of the work in using it is dealing with the additional complexities involved in setting up your build environment in CI to make sure you’re appropriately bundling in any native libraries you depend upon, and linking to them in the correct way. Mypyc’s limitation relative to Cython is a huge advantage here: it doesn’t let you link to other native libraries, so you can always skip the worst step here.

So, for maintainers, you don’t need to maintain a pile of janky VMs on your personal development machine in order to serve your users. For users, nobody needs to deal with the nightmare of setting up the right C compiler on their windows machine, because the wheels are prebuilt. Even users without a compiler who want to contribute new code or debug it can run it with the interpreter locally, and let the cloud handle the complicated compilation steps later. Once again, the fact that you can’t require additional, external C libraries here is a big advantage; it prevents you from making the user’s experience inadvertently worse.

cibuildwheel supports all major operating systems and architectures, and supported versions of Python, and even lets you build wheels for PyPy while you’re at it.9

Putting It All Together

Using Mypyc and cibuildwheel, we, as PyPI package maintainers, can potentially produce an ecosystem of much faster out-of-the-box experiences via prebuilt extension modules, written entirely in Python, which would make the average big Python application with plenty of dependencies feel snappier than expected. This doesn’t have to come with the pain that we have unfortunately come to expect from C extensions, either as maintainers or users.

Another nice thing is that this is not an all-or-nothing proposition. If you try PyPy and it blows up in some obscure way on your code, you have to give up on it unless you want to fully investigate what’s happening. But if you trip over a bug in Mypyc, you can report the bug, drop the module where you’re having the problem from the list of things you’re trying to compile, and move on. You don’t even have to start out by trying to jam your whole project through it; just pick a few key modules to get started, and gradually expand that list over time, as it makes sense for your project.

In a future post, I’ll try to put all of this together myself, and hopefully it’s not going to be embarrassingly difficult and make me eat my words.

Despite not having done that yet, I wanted to put this suggestion out now, to get other folks thinking about getting started with it. For older projects10, retrofitting all the existing infrastructure to put Mypyc in place might be a bit of a challenge. But for new projects starting today, putting this in place when there’s very little code might be as simple as adding a couple of lines to pyproject.toml and copy-pasting some YAML into a Github workflow.

If you’re thinking about making some new open source Python, give Mypyc a try, and see if you can delight some users with lightning speed right out of the box. If you do, let me know how it turns out.

Acknowledgments

Thanks to Donald Stufft, Moshe Zadka, Nelson Elhage, Itamar Turner-Trauring, and David Reid for extensive feedback on this post. As always, any errors or inaccuracies remain my own.


  1. Despite the fact that it is self-described “alpha” software; it’s clearly production-quality enough for Mypy itself to rely upon it, and to have thorough documentation, so if it has bugs that need fixing then it would be good to start discovering them. However, this whole post assumes that you do have good test coverage and you’ll be able to run it over your Mypyc-built artifacts; if you don’t, then this might be too risky. 

  2. I’d love to offer an attribution here, but I have no idea where it came from originally. It’s nearly impossible to search the web these days for things that people were saying in 2005... but as I recall, it grew up as a sort of oral tradition of call-and-response about performance complaints on forums and Usenet. 

  3. At least in the most favorable cases, of course. You can’t do this for everything, but in any sufficiently large C++ system you can always find some fun oversights

  4. as I was, at the time. 

  5. This is not to write off PyO3, which is an excellent tool. It has many uses beyond speed. Beyond the obvious one of “access to libraries in the excellent Rust ecosystem”, one of its important applications is in creating a safer lingua franca among high-level programming languages. If you have a large, complex, polyglot environment with some Ruby, some Java, some Python and some TypeScript, all of which need to share data structures, Rust is a much better option than C for the common libraries that you might need to bind in all of those languages. 

  6. It did for Twisted! We ran our website on absolutely ancient hardware for the better part of a decade and PyPy made it fast enough that nobody really noticed. When we were still using CPython, the site had become barely usable. 

  7. You can get this implementation and a table of the resgults here, on github

  8. I’m not sure that this is a meaningful comparison, but C’s no-cost optimization option of -O3 is a 20% improvement, so we’re in the same ballpark, which is interesting. 

  9. Interestingly, on PyPy, it might actually be faster to upload a pure-Python wheel anyway, since the higher cost of calling into a C module on that platform might negate any benefits of compiling it. But you’ll have to measure it and see. 

  10. not to put too fine a point on it, “like the ones that I maintain” 

Legitimizing Blockchain

Why is advertising blockchain something we should try to stop, rather than ignore?

Yesterday, 1Password made the following announcement:

I am very unhappy about this.

As of this writing, the replies to this announcement are, by my count, roughly 95% paying customers who are furious with them for doing this, 3% scammers who are jubilant that this is popularizing their scamming tool of choice, and about 2% blockchain-enthusiasts expressing confusion as to why everyone is so mad.

Scanning through that 2%’s twitter bios and timelines, I could see content other than memes and shilling, so it seemed at least plausible to me that these people are scam victims who haven’t gotten to the blow-off yet, and their confusion is genuine. Given that “why is everyone so mad” is a much less intense reaction than fury or jubilation, I assume that many others read through some of the vitriol and had this reaction, but then didn’t post anything themselves.

This post is for two audiences: that 2%, genuinely wondering what the big deal is, and also those who have a vague feeling that cryptocurrency is bad, but don’t see the point of making much of a fuss about it.

This is why we should make a fuss about it.


The objection most often raised in the comments went something like this:

This is just a feature that you don’t like; if it’s not for you, just don’t use it. Why yell at 1Password just for making a feature that makes someone else happy?

To begin with, the actual technical feature appears to be something related to auto-filling in browser-extension UI, which is fine. I don’t object to the feature. I don’t even object to features which explicitly help people store cryptocurrency more securely, as a harm reduction measure.

Also, to get this out of the way now: cryptocurrency is a scam. I’m not going to argue the case for that here. Others have made the argument far more exhaustively, and you can read literally hundreds of pages and watch hours of video explaining why by clicking here.

The issue is with the co-marketing effort: the fact that 1Password is using their well-respected brand to help advertise and legitimize scam-facilitation technology like Solana and Phantom.

Even if we were to accept all this, it’s a scam, 1Password is marketing it, etc, my hypothetical blockchain-curious interlocutor here might further object:

What’s the big deal about legitimizing these things, even if they are fraud? Surely you can just not get defrauded, and ignore the marketing?

That’s true, but it also misses the point: legitimizing and promoting these things does various kinds of harm.

More broadly, although I’m writing about 1Password’s specific announcement here, and a small amount of the reasoning will be specific to password management tools, most of the concerns I’ll describe are fairly general to any company promoting or co-marketing with cryptocurrency, and thus hopefully this post will serve for future instances where we should tell some other company to stop supporting blockchains as well.

So with all that out of the way, here are some of the harms that one might be concerned about, from the least selfish concern to the most.


Concern #1: the well-being of others

I don’t know how to explain to you that you should care about other people, but if you do care about other people, this could hurt them.

First and foremost, the entire scam of cryptocurrency rests upon making people believe that the assets are worth something. Most people are not steeped in the technical minutiae of blockchains, and tend to trust things based on institutional reputation. 1Password has a strong brand, related to information security, and they’re saying that cryptocurrencies are good, so it’s likely to convince a nonzero number of people to put their money into this technology that has enormous non-obvious risks. They could easily lose everything.

Advertising 1Password in this way additionally encourages users to maintain custody of their own blockchain assets on their own devices. Doing so with 1Password is considerably less risky than it might be otherwise, so if this were to only reach people who were already planning to store their wallets on their own computers, then great.

However, this might encourage users who had not previously thought to look at cryptocurrency at all to do so, and if they found it via 1Password they might start using 1Password to store their first few secrets. Storing them in this way, although less risky, is still unreasonably risky, given the lack of any kind of safety mechanisms on blockchain-backed transactions. Even if they’re savvy enough not to get scammed, nobody is savvy enough not to get hacked, particularly by sophisticated technical attacks which are worth leveraging against high-value targets like people with expensive crypto wallets on their computers.

To be clear, crypto exchanges are, on average, extremely bad at the job of not getting their users money stolen, but individual users are likely to be even worse at that job.

Concern #2: economic damage

If you don’t care about other people much, but you still care about living in a functioning society, then the promotion of blockchain based financial instruments is a huge destabilization risk. As Dan Olson explains in the devastating video essay / documentary Line Goes Up, blockchain-based financial instruments share a lot of extremely concerning properties that made mortgage-backed securities and collateralized debt obligations so financially toxic in the 2008 crash. Large-scale adoption of these things could lead to a similar crisis, or even worse, a global deflationary spiral in the style of the one that caused the great depression, setting off the kind of economic damage that could result in mass famine and mass death.

Of course, any individual company or celebrity advertising crypto is not going to trigger an immediate economic collapse. Each of these is a snowflake in an avalanche. I have no illusions that convincing just 1Password to stop this is going to turn the tide of the entire blockchain catastrophe that is unfolding all around us, or indeed that my one little post here is going to make the decisive difference between, 1Password stopping vs. not.

But that’s exactly why I’m trying to persuade you, dear reader, that this is a big deal and we should all try to work together to stop it.

Concern #3: environmental damage

While this specific blockchain is “greener” than others, but given the huge proportion of cryptocurrency generally that is backed by electrical waste, and the cultural and technical incentives that make trading one blockchain asset for another more common than cashing out to dollars, it’s still a legitimate concern that promoting blockchain in general will promote environmental destruction indirectly.

Furthermore, the way that Solana is less energy-intensive than other blockchains is by using proof-of-stake, so there’s a sliding scale here between economic and environmental damage, given that proof-of-stake is designed to accelerate wealth accumulation among non-productive participants, and thereby encourages hoarding. So the reduction in environmental damage just makes the previous point even worse.

Concern #4: increased targeting risk

Even if you’re a full blown sociopath with no concern for others and an iron-clad confidence that you can navigate the collapse of the financial system without any harm to you personally, there is still a pretty big negative here: increased risk from threat actors. Even if you like and use blockchain, and want to use this feature, this risk still affects you.

If 1Password happened to have some features that blockchain nerds could use to store their secrets, then attackers might have some interest in breaking in to 1Password, and could possibly work on tools to do so. That’s the risk of existing on the Internet at all. But if 1Password loudly advertises, repeatedly, that they are will be integrating with a variety of cryptocurrency providers, then this will let attackers know that 1Password is the preferred cryptocurrency storage mechanism.

This further means that attackers will start trying to figure out ways to target 1Password users, on the assumption that we’re more likely to have crypto assets lying around on our filesystems; not only developing tools to break in to 1Password but developing tools to fingerprint users who have the extension installed, who have accounts on the service, whose emails show up on the forum, etc.

Now, of course, 1Password users keep plenty of high-value information inside 1Password already; that’s the whole point. But cryptocurrency is special because of the irreversible nature of transactions, and the immediacy of the benefit to cybercriminals specifically.

If you steal all of someone’s bank passwords, you could potentially get a bunch of their money, but it is expensive and risky for the criminals. The transactions can be traced directly to actual human account holders immediately; anti-money-laundering regulations mean that this can usually be accomplished even across international borders. Transfers can be reversed.

This discrepancy between real money and cryptocurrency is exactly why ransomware was created by cryptocurrency. It makes cryptocurrency attractive specifically to the kinds of people who have expertise and resources to mount wide-spectrum digital attacks against whole populations.

Of course, if they develop tools to fingerprint and hack 1Password users, but they don’t luck out and find easy-to-steal crypto on your computer, they might as well try to steal other things of value, like your identity, credit information, and so on. These are higher-risk, but now that they’ve built all that infrastructure and hacked all these machines, there’s a big sunk cost that makes it more worthwhile.

Please Stop

I really hope that 1Password abandons this destructive scheme. Even if they fully walk this back, I will still find it much harder to recommend their product in the future; there will need to be some active effort to repair trust with their user community. If I’ve convinced you of the problems here, please let them know as a reply to the tweet, the email linked from their blog post, their community forum, or the Reddit post of the announcement, so that they can get a clear signal that this is unacceptable.