One of the wonderful things about Python is the ease with which you can start writing a script - just drop some code into a .py file, and run python my_file.py. Similarly it’s easy to get started with modularity: split my_file.py into my_app.py and my_lib.py, and you can import my_lib from my_app.py and start organizing your code into modules.

However, the details of the machinery that makes this work have some surprising, and sometimes very security-critical consequences: the more convenient it is for you to execute code from different locations, the more opportunities an attacker has to execute it as well...

Python needs a safe space to load code from

Here are three critical assumptions embedded in Python’s security model:

1. Every entry on sys.path is assumed to be a secure location from which it is safe to execute arbitrary code.
2. The directory where the “main script” is located is always on sys.path.
3. When invoking python directly, the current directory is treated as the “main script” location, even when passing the -c or -m options. **

If you’re running a Python application that’s been installed properly on your computer, the only location outside of your Python install or virtualenv that will be automatically added to your sys.path (by default) is the location where the main executable, or script, is installed.

For example, if you have pip installed in /usr/bin, and you run /usr/bin/pip, then only /usr/bin will be added to sys.path by this feature. Anything that can write files to that /usr/bin can already make you, or your system, run stuff, so it’s a pretty safe place. (Consider what would happen if your ls executable got replaced with something nasty.)

However, one emerging convention is to prefer calling /path/to/python -m pip in order to avoid the complexities of setting up $PATH properly, and to avoid dealing with divergent documentation of how scripts are installed on Windows (usually as .exe files these days, rather than .py files).

This is fine — as long as you trust that you’re the only one putting files into the places you can import from — including your working directory.

Your “Downloads” folder isn’t safe

As the category of attacks with the name “DLL Planting” indicates, there are many ways that browsers (and sometimes other software) can be tricked into putting files with arbitrary filenames into the Downloads folder, without user interaction.

Browsers are starting to take this class of vulnerability more seriously, and adding various mitigations to avoid allowing sites to surreptitiously drop files in your downloads folder when you visit them.¹

Even with mitigations though, it will be hard to stamp this out entirely: for example, the Content-Disposition HTTP header’s filename* parameter exists entirely to allow the the site to choose the filename that it downloads to.

Composing the attack

You’ve made a habit of python -m pip to install stuff. You download a Python package from a totally trustworthy website that, for whatever reason, has a Python wheel by direct download instead of on PyPI. Maybe it’s internal, maybe it’s a pre-release; whatever. So you download totally-legit-package.whl, and then:

1
2

~$ cd Downloads
~/Downloads$ python -m pip install ./totally-legit-package.whl

This seems like a reasonable thing to do, but unbeknownst to you, two weeks ago, a completely different site you visited had some XSS JavaScript on it that downloaded a pip.py with some malware in it into your downloads folder.

Boom.

Demonstrating it

Here’s a quick demonstration of the attack:

1
2
3
4
5

~$ mkdir attacker_dir
~$ cd attacker_dir
~/attacker_dir$ echo 'print("lol ur pwnt")' > pip.py
~/attacker_dir$ python -m pip install requests
lol ur pwnt

PYTHONPATH surprises

Just a few paragraphs ago, I said:

If you’re running a Python application that’s been installed properly on your computer, the only location outside of your Python install or virtualenv that will be automatically added to your sys.path (by default) is the location where the main executable, or script, is installed.

So what is that parenthetical “by default” doing there? What other directories might be added?

Anything entries on your $PYTHONPATH environment variable. You wouldn’t put your current directory on $PYTHONPATH, would you?

Unfortunately, there’s one common way that you might have done so by accident.

Let’s simulate a “vulnerable” Python application:

1
2
3
4
5

# tool.py
try:
    import optional_extra
except ImportError:
    print("extra not found, that's fine")

Make 2 directories: install_dir and attacker_dir. Drop this in install_dir. Then, cd attacker_dir and put our sophisticated malware there, under the name used by tool.py:

1
2

# optional_extra.py
print("lol ur pwnt")

Finally, let’s run it:

1
2

~/attacker_dir$ python ../install_dir/tool.py
extra not found, that's fine

So far, so good.

But, here’s the common mistake. Most places that still recommend PYTHONPATH recommend adding things to it like so:

1

export PYTHONPATH="/new/useful/stuff:$PYTHONPATH";

Intuitively, this makes sense; if you’re adding project X to your $PYTHONPATH, maybe project Y had already added something, maybe not; you never want to blow it away and replace what other parts of your shell startup might have done with it, especially if you’re writing documentation that lots of different people will use.

But this idiom has a critical flaw: the first time it’s invoked, if $PYTHONPATH was previously either empty or un-set, this then includes an empty string, which resolves to the current directory. Let’s try it:

1
2
3

~/attacker_dir$ export PYTHONPATH="/a/perfectly/safe/place:$PYTHONPATH";
~/attacker_dir$ python ../install_dir/tool.py
lol ur pwnt

Oh no! Well, just to be safe, let’s empty out $PYTHONPATH and try it again:

1
2
3

~/attacker_dir$ export PYTHONPATH="";
~/attacker_dir$ python ../install_dir/tool.py
lol ur pwnt

Still not safe!

What’s happening here is that if PYTHONPATH is empty, that is not the same thing as it being unset. From within Python, this is the difference between os.environ.get("PYTHONPATH") == "" and os.environ.get("PYTHONPATH") == None.

If you want to be sure you’ve cleared $PYTHONPATH from a shell (or somewhere in a shell startup), you need to use the unset command:

1
2

~/attacker_dir$ python ../install_dir/tool.py
extra not found, that's fine

Setting PYTHONPATH used to be the most common way to set up a Python development environment; hopefully it’s mostly fallen out of favor, with virtualenvs serving this need better. If you’ve got an old shell configuration that still sets a $PYTHONPATH that you don’t need any more, this is a good opportunity to go ahead and delete it.

However, if you do need an idiom for “appending to” PYTHONPATH in a shell startup, use this technique:

1
2

export PYTHONPATH="${PYTHONPATH:+${PYTHONPATH}:}new_entry_1"
export PYTHONPATH="${PYTHONPATH:+${PYTHONPATH}:}new_entry_2"

In both bash and zsh, this results in

1
2

$ echo "${PYTHONPATH}"
new_entry_1:new_entry_2

with no extra colons or blank entries on your $PYTHONPATH variable now.

Finally: if you’re still using $PYTHONPATH, be sure to always use absolute paths!

Related risks

There are a bunch of variant unsafe behaviors related to inspecting files in your Downloads folder by doing anything interactive with Python. Other risky activities:

• Running python ~/Downloads/anything.py (even if anything.py is itself safe) from anywhere - as it will add your downloads folder to sys.path by virtue of anything.py’s location.
• Jupyter Notebook puts the directory that the notebook is in onto sys.path, just like Python puts the script directory there. So jupyter notebook ~/Downloads/anything.ipynb is just as dangerous as python ~/Downloads/anything.py.

Get those scripts and notebooks out of your downloads folder before you run ’em!

But cd Downloads and then doing anything interactive remains a problem too:

• Running a python -c command that includes an import statement while in your ~/Downloads folder
• Running python interactively and importing anything while in your ~/Downloads folder

Remember that ~/Downloads/ isn’t special; it’s just one place where unexpected files with attacker-chosen filenames might sneak in. Be on the lookout for other locations where this is true. For example, if you’re administering a server where the public can upload files, make extra sure that neither your application nor any administrator who might run python ever does cd public_uploads.

Maybe consider changing the code that handles uploads to mangle file names to put a .uploaded at the end, avoiding the risk of a .py file getting uploaded and executed accidentally.

Mitigations

If you have tools written in Python that you want to use while in your downloads folder, make a habit of preferring typing the path to the script (/path/to/venv/bin/pip) rather than the module (/path/to/venv/bin/python -m pip).

In general, just avoid ever having ~/Downloads as your current working directory, and move any software you want to use to a more appropriate location before launching it.

It’s important to understand where Python gets the code that it’s going to be executing. Giving someone the ability to execute even one line of arbitrary Python is equivalent to giving them full control over your computer!

Why I wrote this article

When writing a “tips and tricks” article like this about security, it’s very easy to imply that I, the author, am very clever for knowing this weird bunch of trivia, and the only way for you, the reader, to stay safe, is to memorize a huge pile of equally esoteric stuff and constantly be thinking about it. Indeed, a previous draft of this post inadvertently did just that. But that’s a really terrible idea and not one that I want to have any part in propagating.

So if I’m not trying to say that, then why post about it? I’ll explain.

Over many years of using Python, I’ve infrequently, but regularly, seen users confused about the locations that Python loads code from. One variety of this confusion is when people put their first program that uses Twisted into a file called twisted.py. That shadows the import of the library, breaking everything. Another manifestation of this confusion is a slow trickle of confused security reports where a researcher drops a module into a location where Python is documented to load code from — like the current directory in the scenarios described above — and then load it, thinking that this reflects an exploit because it’s executing arbitrary code.

Any confusion like this — even if the system in question is “behaving as intended”, and can’t readily be changed — is a vulnerability that an attacker can exploit.

System administrators and developers are high-value targets in the world of cybercrime. If you hack a user, you get that user’s data; but if you hack an admin or a dev, and you do it right, you could get access to thousands of users whose systems are under the administrator’s control or even millions of users who use the developers’ software.

Therefore, while “just be more careful all the time” is not a sustainable recipe for safety, to some extent, those of us acting on our users’ behalf do have a greater obligation to be more careful. At least, we should be informed about the behavior of our tools. Developer tools, like Python, are inevitably power tools which may require more care and precision than the average application.

Nothing I’ve described above is a “bug” or an “exploit”, exactly; I don’t think that the developers of Python or Jupyter have done anything wrong; the system works the way it’s designed and the way it’s designed makes sense. I personally do not have any great ideas for how things could be changed without removing a ton of power from Python.

One of my favorite safety inventions is the SawStop. Nothing was wrong with the way table saws worked before its invention; they were extremely dangerous tools that performed an important industrial function. A lot of very useful and important things were made with table saws. Yet, it was also true that table saws were responsible for a disproportionate share of wood-shop accidents, and, in particular, lost fingers. Despite plenty of care taken by experienced and safety-conscious carpenters, the SawStop still saves many fingers every year.

So by highlighting this potential danger I also hope to provoke some thinking among some enterprising security engineers out there. What might be the SawStop of arbitrary code execution for interactive interpreters? What invention might be able to prevent some of the scenarios I describe below without significantly diminishing the power of tools like Python?

Stay safe out there, friends.

Acknowledgments

Thanks very much to Paul Ganssle, Nathaniel J. Smith, Itamar Turner-Trauring and Nelson Elhage for substantial feedback on earlier drafts of this post.

Any errors remain my own.

I suffer from ADHD.

Photo by Taylor Young on Unsplash

Update 2021-08-22: Almost exactly a year after this post was written, I sought and received a clinical neuropsychiatric diagnosis of ADHD (predominantly inattentive, sluggish cognitive tempo), so I’m no longer self-diagnosed.

I want to be clear: when I say I suffer from this disorder, I am making a self-diagnosis. I’ve obliquely referred to suffering from ADHD in previous posts, but rarely at any length. The main reason for my avoidance of the topic is that it still makes me super uncomfortable to write publicly about a “self-diagnosis”, since there’s a tremendous amount of Internet quackery thanks to amateur diagnosticians.

This despite the fact that I’ve known for the past 15 years that I have ADHD.

I am absolutely not trying to set myself up as a maverick unlicensed freelance psychiatrist here. If you think you might have ADHD, or any other ailment, whether mental or physical, call your primary care physician. Don’t email me.

At the same time, for me, this diagnosis is not really ambiguous or in a gray area. This is me looking down and noticing I’ve only got one arm, and diagnosing myself as a one-armed person. I’ve taken numerous ADHD screening questionnaires and reliably scored well into the range of “there is no ambiguity whatsoever, you absolutely have ADHD”, so I feel confident to describe myself as having it.

Terminology aside, this post is about a set of cognitive and metacognitive issues that I have, and some tools that I found useful to remedy them. I think others might find those same tools useful in similar situations. So if you’re also uncomfortable with the inherently unreliable nature of self-diagnosis, or the clinical specificity of the term “ADHD” — and I absolutely don’t blame you if you are — I invite you to read “ADHD” as a shorthand for some character traits that I informally believe fit that label, and not a robust clinical analysis of myself or anyone else.

With that extended disclaimer out of the way, I’ll get started on the post itself; and where better to do that than at the start of my own challenges.

The ‘Laziness’ model

Photo by Zosia Korcz on Unsplash

Throughout my childhood, I was labeled an “underachiever”. I performed well on tests and didn’t do homework. I was frequently told by adults — especially my teachers — that I was “brilliant” but “lazy”.

Was I lazy? Is there even such a thing as “laziness”? Here’s a spoiler for you — “no”¹ — but I didn’t know that at the time. All I knew was that I couldn’t seem to do certain things — boring things: homework, long division, and cleaning up my room, for a few examples. I couldn’t seem to do the things that my peers found routine and trivial.

This is a common enough experience that it shows up clearly even in systematic reviews and meta-analyses of adult sufferers of ADHD. Everybody tells you you’re lazy, and so you believe it. It sure looks like laziness from the outside!

In retrospect, that’s the interesting problem with this false diagnosis: “from the outside”. Assuming for the moment that laziness does in fact exist and is a salient character flaw, what would the experience of the interiority of such laziness actually feel like?

It seems unlikely that it would feel like I what I actually felt at the time:

1. Frequently, suddenly remembering, in contexts where it wouldn’t help — walking to school, in an unrelated class, while walking to work — that I had to Do The Thing.

2. Anxiously, yearningly, often desperately wishing I could Do The Thing.

3. Trying to Do The Thing at the responsible time, finding that my mind would wander and I would lose several hours of time... sitting for hours, literally bored to tears, while I attempted and failed to Do The Thing.

4. At long last, finally managing to start. Once I was truly exhausted and starting to panic, I’d drink a gallon of heavily-caffeinated and very sugary soda at 2 in the morning and finally finally find that I suddenly had the ability to Do The Thing, and white-knuckle my way through an all-nighter to finish The Thing. (This step was more common after I got to my late teens; before that, The Thing just wouldn’t get Done.)

Sitting up night after night destroying my mental and physical health, depriving myself of sleep, focusing with every ounce of my will on tasks that I absolutely hated doing but was forcing myself to complete at all costs: it doesn’t seem to line up with the popular conception of what “laziness” might be like! Yet, I absolutely believed that I was lazy. If I were not lazy, surely Doing The Thing wouldn’t be so difficult!

I took pains at the start of this post to point out that mental health diagnosis is usually best left to professionals. I think that at this point in the story I should emphasize that “I’m lazy” is also itself a self-diagnosis, and — at least in every case where I’ve ever heard it used — a much worse one than “I have ADHD”.

If you are not a licensed psychologist or psychiatrist, any time you decide with certainty that someone (even yourself!) has an intrinsic, persistent character flaw, you’re effectively diagnosing them. If you decide that they’re inherently lazy, or selfish, or arrogant, you’re effectively diagnosing them with a sort of personality disorder of your own invention.

So, although I didn’t see it at the time, laziness didn’t seem to describe me terribly well. What description fits better?

The ‘Attention Deficit’ model

Photo by Tom Bradley on Unsplash

In my late 20s, my Uncle Joel gave me a gift that changed my life: the book “Driven to Distraction: Recognizing and Coping with Attention Deficit Disorder”² by Edward M. Hallowell. The life-changing aspect of this book was not so much that it showed that there were other people “like me”, or that my problem had a name, but that it gave me a different, and more accurately predictive, model to understand my own behavior.

In other words, it allowed me to see — for the first time — that the scarcest resource limiting my efficacy wasn’t the will to do the work, but rather the ability to focus. With this enhanced understanding, I could select a more effective strategy for dealing with the problem.

I did select such a strategy! It worked very well — albeit with some caveats. I’ll get to those in a moment.

Although my limiting factor was the ability to pay attention, the problem that prevented me from recognizing this was one of metacognition — the way I was thinking about how I think.

My early model of my own mind was that I was a lazy person who just needed to do what I had assumed everyone else must be doing: forcing myself to do the tasks that I was having trouble completing. If I really wanted to get them done, then what possible other reason could there be for me to not do them?

The ‘laziness’ model didn’t generate particularly good predictions. For any given project at school, it would predict that I would not try very hard to do it, since the very dictionary definition of ‘lazy’ is “unwilling to work or use energy”. The observed behavior, by contrast, was constant, panicked, intense (albeit failed, or at least highly inefficient) uses of significant amounts of energy.

The main reason to have a model of a thing is to make predictions about that thing. If the predictions that a model gives you are consistently wrong, then the model isn’t directly useful. At that point, it’s time to discard it and find a better one. At the very least, it’s time to revise the model in question until it starts giving you more accurate, actionable information.

The ‘laziness’ model is wrong, but worse than that, it’s harmful. What it routinely predicts, regardless of context, is that I need more negative self-talk, more ‘motivation’ in the form of vicious self-criticism, more forcing myself to “just do it”. All of these things, particularly when performed habitually, cause real, significant harm.

If I gave myself the most negative self-talk I could muster, the most vicious criticism, and really put Maximum Effort into forcing myself to do the thing I wanted done... if it didn’t work, of course that just meant that I needed to engage in even more self-abuse! I could always try harder!

This is the worst way that a model can be inaccurate: an unfalsifiable, self-reinforcing prediction. I could never demonstrate to myself that I’d really been as unkind to myself as was possible; there was always room for escalation. Psychologically, it’s also the worst kind of behavioral advice, which is the kind that generates a self-reinforcing negative feedback loop.

Once I started putting my newfound knowledge into practice, the difference between interventions predicated on an understanding of the problem as “lack of usable attention span” and those based on “lack of willpower” was night and day. I stopped trying to white-knuckle my way through all of my challenges and developed non-judgmental ways to remind myself to do things.

I knew that I, personally, was never going to spontaneously remember to do things at the right time, so I developed ways of letting computers remind me. I knew that I’d never be able to stick with routine, repetitive tasks, so I made a unified list of all the tedious administrative tasks I need to perform. I can’t keep important dates and times in mind, so I rely completely upon my calendar.

Even given these successes, “it worked!” is a colossal oversimplification. Today, it’s about 15 years later, and I’m still sifting through the psychological rubble wrought by the destructive, maladaptive coping mechanisms that I just described, and still trying to find better ways to remain effective when I’m feeling distracted... which is most of the time.

Simply having a better model at the coarsest level is just the first step. Instantiating that model in a working, fleshed out technological system is a ton of work in its own right.³ But it’s work that starts having little successes, which is a lot easier to build on and maintain momentum with than the same failure repeated day after day.

Given that I was starting — nearly from scratch — at 25, and had a lifetime worth of bad habits to unlearn, constructing a workable system that addressed my personal organizational needs still took the better part of a decade.

So as I move into the next, slightly more prescriptive section here, I don’t want to give anybody the idea that I think this is easy.

Don’t give up!

Listen up, Simon. Don’t believe in yourself. Believe in me! Believe in the Kamina who believes in you!

Kamina, Episode 1,
Tengen Toppa Gurren Lagann

At the start of this post, I specifically mentioned that I hadn’t wanted to write at length about ADHD due to my discomfort with self-diagnosis. So, you might be wondering: what was it that overcame this resistance and prompted me to finally write about my own experiences with ADHD?

The original inspiration was a pattern of complaints about suffering from ADHD I see periodically — mainly on Twitter — that look roughly like this:

• “ADHD means never being on time for a meeting and having no excuse, forever.”
• “It’s great to have ADHD and never be able to complete a routine task. Sigh.”
• “I can’t take out the trash and my roommates just can’t understand that this is just part of who I am and I will never get better.”
• “Why can’t neurotypicals understand that I’m just never going to “get stuff done” like they can. It’s exhausting.”

These are paraphrased and anonymized on purpose; I really don’t want to direct any negative attention towards someone specific, particularly someone just venting about struggles.

Of course, no blog post in mid-2020 would be complete without some reference to the ... situation. The original inspiration for this post predates the dawn of the new hell-world we all now inhabit, but, to say the least, COVID-19⁴ has presented some new challenges to the coping mechanisms I’m writing about here. (Still, I know that I’m considerably better off than the average American in this mess.)

The message I’m trying to get across here is hopeful — others suffering with executive-function deficits similar to mine might be able to do what I did and fix a lot of their problems with this one weird trick! — and the constant drumbeat of despair all around us right now makes that sort of message feel more urgent.

Posts like the ones I described above seem to represent a recurring pattern of despair, and they make me sad. Not because I can’t identify with them; I have absolutely had these feelings. Not even because they’re wrong, exactly: it really is harder for folks with ADHD to handle some of these situations, and the struggle really is lifelong.

They make me sad because they’re expressing a fatalistic perspective; a fixed mindset⁵ that precludes any hope of future improvement. The through line that I have seen from all of these posts is a familiar, specific kind of despair; a thought I’ve had myself multiple times:

When somebody that I care about asks me, ‘Can you do the dishes later?’, I want to say ‘yes’ and have them believe me. I want to be able to believe myself, and I don’t think I will ever be able to.

Unlike myself when I was younger, the authors of these posts already have a name for their problem: ADHD. Sometimes they’ve even tried some amount of therapy or even medication.

Even so, they’re still buying in to the maladaptive strategy of “just try harder”. Since they already know that ADHD is, at least in part, a structural brain difference, they despair of ever being able to actually do that though, which leaves “giving up” as the only viable strategy.

Don’t give up! I believe in you!

Different problems, different tools

I have had another lifelong problem since when I was young: I am severely nearsighted. Yet, I never developed any psychological hangups around that; nobody ever told me that I needed to buckle down and just squint harder. This problem was socially quite well understood, so… I got glasses. Then I could see, as long as I consistently used those glasses.

Nobody ever expected me to be able to see without glasses.

Photo by NordWood Themes on Unsplash

Calendars, to-do lists, and systems like Getting Things Done are the corrective lenses for the ADHD brain.⁶.

If a to-do list is a corrective lens for ADHD, one of the major issues around understanding how to use it is that the mass-market literature around to-do lists assumes a certain level of neurotypicality. Assistive devices may frequently be useful to non-disabled people, but their relationship to and use of such affordances is very different.

Through the Looking-Glass ...

Let’s stretch this lens metaphor into absurdity.

In our metaphorical world, ADHD is myopia, and so most — or at least many — folks are “sight-typical”. Productivity systems are our “lenses”.

If nearsightedness were as poorly understood as ADHD, and you were nearsighted, you wouldn’t be able to pop on down to Lenscrafters and pick up a pair of spectacles. You might realize that the problem was with your eyes, and think, “lenses might help me see farther”. Many kinds of lenses might be commercially available in such a world! Lenses for telescopes, cameras, microscopes...

The way that someone with 20/20 vision might use a lens to see farther is to use a telescope to see something really far away. But you, my hypothetically-nearsighted friend, don’t need a powerful zoom lens to take surveillance photographs from a helicopter. Even if you could make such lenses work to correct your vision, you wouldn’t want to carry a pair of 2-kilogram DSLR zoom lenses everywhere you go. You want eyeglasses, which are something different.

Photo by James Bold on Unsplash

The lenses in eyeglasses are — while operating on fundamentally the same principles of optics as the lenses in a telescope or a microscope — constructed and packaged in a completely different way. But most importantly, the way you use them is to wear them every day, not to deploy them on special occasions in the rare event where you need to do something extreme, but all the time, every day, in the same way.

... and What I Found There

Photo by Glenn Carstens-Peters on Unsplash

A person with nominal executive function might use the occasional free-floating to-do list to track a big, complex project with a lot of small interrelated tasks. Most folks in the modern information-driven economy routinely need to do projects that are too complex to easily memorize all the required steps. Even doing your own personal taxes has enough steps to require at least a little bit of tracking.

Such a person could make a to-do list for that one project — their telescope, if you will — put it in a place where they’d remember to look at it when they’re working on that project, and then remember to check things off when they’re done.

They could have one to-do list on the fridge for groceries, a note on their phone for stuff to get for their spouse, and a wiki page outlining some tasks at work. They would probably have enough free-floating executive function to remember which list maps to which project and when each project is relevant, and remember to check each one at the appropriate time.

I spent a lot of time trying to make disconnected to-do lists like this work for me. They never have. Even when I’m feeling particularly productive there is a cycle of list-generation, that goes like this:

1. When I want to work on the project in question, I can’t remember where the to-do list is, but I need to figure out what I need to do again.

2. So I go and write a new to-do list, spend a bunch of time rewriting the one I’d already written but can’t quickly find. Then I do some work on the project, check off a few things, and put the list away.

3. Later, I’ll find both lists, both half checked off, and now I waste a bunch of time trying to figure out which one is the right one.

4. Repeat this process a few times, and now I have a dozen lists. The lists themselves start generating more work than the actual project, because now I am constantly re-making and finding lists, trying to figure out which one is the most up to date.

This is the simplest case, but the real problem happens at a higher level: one of the biggest problems caused by any executive function deficit like ADHD is the difficulty of task initiation.

The more irrelevant distractions I can see while I’m trying to work out what to do next, the harder that decision becomes. And there’s nothing quite so distracting as the detritus of a thousand half-finished to-do lists.

One List To Rule Them All

What I’ve found works for me is a single, primary to-do list that I can obsessively check in with every minute of every day, which subsumes every other list related to every other project in my life.

I’m hardly the only person to have this insight — if you start engaging with the “productivity” noosphere, reading all the books, listening to the podcasts, this is a recurring theme. You don’t just have an ‘app’ or a ‘list’, you have to have a System. It has to be reliable; you have to know you’re going to keep checking it, or it’s worthless for storing your commitments. But unfortunately this is frequently buried under a lot of other technical complexity about the fiddly details of how to set up one system or the other. It’s very easy to miss the forest for the trees.

Having ADHD means that I routinely forget what I’ve decided to do over the course of only a minute or two after I’ve decided to do it. Just this week, I had to remind myself no fewer than three times to write down “buy more olive oil” because I kept remembering that we were running low when I was in the kitchen and by the time I finished washing my hands to put it into my phone I’d already forgotten why I did that and went back to making dinner.

I need to write everything down. I’m not going to remember five or six, or even two or three places to check for what to do next. I need to have one place to check what comes next, and then build the habit of constantly going back to it, both to add new things and to see what needs to be done.

Technology can help. Technology might even be necessary — it is for me.

But if you’re considering trying this out for the first time, be mindful that piles of to-do apps can be just as distracting as piles of paper. The important thing is to clearly, singularly decide on the one place which is the ‘root’ of your task tracking system.

You can even do this with a pen and paper. Carry the same, single notebook with you everywhere, and make it absolutely clear that it is your primary list, which is where you have to put any references to other lists. Some people have a lot more success with something tactile, to engage all the senses.

For me personally, the high-tech portion of this strategy is indispensable. I use a combination of OmniFocus for things that have to be done and Apple’s built-in calendar application for places I have to be at a particular time.⁷

OmniFocus⁸ defines the core gameplay loop of my life. Rather than having to cultivate and retain an elaborate series of interlocking habits and rituals to remain functional, I have a single root habit which triggers every other habit.

That habit? Consulting the unified “what should I do next” perspective in OmniFocus. Every time I am even marginally distracted, I check that view again.

Any time I have trouble initiating a task, I start breaking down the top task in that list into smaller and smaller “next physical action”. I don’t even rely on myself to do this; since I know I’ll forget to break things down, I frequently make tasks that look like this:

• thing I want to do
• plan the thing I want to do
  • break down the planning task into tiny actions and write them down here
  • break down the task itself into tiny actions and write them down here

To reduce distraction, I routinely close down any windows that are not necessary for whatever I’m currently working on. Particularly, I routinely sweep to get rid of browser tabs, asking (as I would with an email) “does this window represent a task I should do?”. If yes, it goes in the task system, if no, I close it so it won’t distract me further.

To facilitate this clean-up, on every computer that I use, I have a global hot-key set up to turn the thing that I’m looking at — some selected text, an image, an email message, a browser tab, a chat message at work — into a task that I can look at later.

Everything I have to do on a regular basis is in this system as a recurring task; for example:

• taking out the trash
• doing the dishes
• logging in to Jira at work to look for assigned tasks
• checking my email
• brushing my teeth

Yes, even basic personal hygiene is in here. Not because I’ll necessarily forget, or that it takes a lot of energy, but I don’t want to waste one iota of brainpower I could be devoting to my current task to worrying about whether I might need to do something else later. If I don’t see ‘brush teeth’ in my “what should I do next” view, then I know, with certainty, that I don’t need to be thinking about tooth-brushing right now.

The “what should I do next” view is available on all of my computers, on my tablet, on my phone, and it even dominates my watch-face; I check it more often than I check the time:

No single feature is a hard requirement of my system; I could get along without any one of them in a pinch. However, the way that they combine to constantly reinforce what the next thing I need to do is in any given context, at any given time, means that I need to expend less energy trying to consciously hang on to all the context.

Limitations and Risks

I don’t want to give an overly rosy view of this strategy. Getting a single unified to-do system that works for you is not the same as getting a brain that can remember to do stuff. So here are some caveats:

1. Implementing and maintaining such a system is never easy. It just takes tasks like ‘making sure I renew my passport before I need to travel’, ‘show up on time for the meeting’ and ‘buy a gift at least a week before the wedding’ from totally impossible to possible to do at least somewhat reliably with a sustainable level of effort. The main thing that I believe is possible for everyone is being able to commit to simple future tasks.
2. Building enough data about one’s own habits and procrastination triggers also takes time, and to make such a system effective, one needs to do that work as well. (A passive time-tracking tool like Screen Time on your phone or RescueTime on your workstation can be quite illuminating — and surprising.)
3. The initial wave of relief I felt when I started tracking tasks masked a gradual increase in my general anxiety over time. Checking and re-checking the ‘what to do next’ list can become a bit of an anxious compulsion, a safety behavior that doesn’t always help me plan my day. As one builds the habit of routinely checking the list, it’s important to avoid developing constant anxiety about the list as the only motivation to do so.
4. Similarly, it is important to learn to under-commit. Not only does one need to avoid putting an unrealistic amount of stuff into the system, everybody (but especially everybody with ADHD!) needs non-trivial chunks of unstructured, unplanned time, where the system will clearly say ‘nothing to do now, just relax’. The “poor self-observation” and “time blindness” symptoms of ADHD ensure that properly estimating things before committing is a constant challenge that never really goes away either.
5. This strategy definitely won’t be sufficient for some folks. ADHD is a spectrum and there’s no precise mechanism to calibrate where you are on it. Some folks will respond really well to this strategy, some folks will need medication before it helps to a useful degree.

Finishing up (about finishing up)

If you’re suffering from ADHD and despairing that you will never finish a task or be on time to an appointment: you can. It’s possible to do it at least pretty reliably. I believe if you commit to one and only one task tracking system, and consistently use it every single day, all the time, you can commit to tasks and get them done.

If you do it consistently enough, it will eventually become muscle memory, and not something you need to consciously remember to do every day.

It’s still never going to be easy to Do The Thing, even if your digital brain can perfectly remember what The Thing is right now.

At the very least, it was possible for me to learn to trust myself when I say that I will do something in the future, by designing a system around my own limited attention, and if I can do it, I think you can too.

Acknowledgments

This was a big one! I’d like to particularly thank my Uncle Joel, without whom this post (and many of my other achievements) would not be possible for the reasons described above, as well as Moshe Zadka, Amber Brown, Tom Most, and Eevee for extensive feedback on previous drafts of this post.

Additionally, I’d like to thank David Reid for introducing me to many of the tools and techniques that I still use every day, and Cory Benfield, Jonathan Lange, and Hynek Schlawack for many illuminating conversations over the years about the specifics and detailed mechanics of the tools whose use I describe in this post.

Any errors, of course, remain my own.

Get it?
Quack quack quack quack
Quack quack quack quack

Weird Al Yancovic,
“I Want A New Duck”

Mypy makes most things better

Mypy is a static type checker for Python. If you’re not already familiar, you should check it out; it’s rapidly becoming a standard for Python projects. All the cool kids are doing it. With Mypy, you get all the benefits of high-level dynamic typing for rapid experimentation, and all the benefits of rigorous type checking to complement your tests and improve reliability.¹ The best of both worlds!

Mypy can change how you write Python code. In most cases, this is for the better. For example, I have opined on numerous occasions about how bad None is. But this can significantly change with Mypy. Now, when you return None, you can say -> Optional[str] and rest assured that all your callers will be quickly, statically checked for places where they might encounter an AttributeError on that None, which makes this a more appealing option than risking raising a runtime exception (which Mypy can’t check).

But sometimes, things can get worse

But in some cases, as you add type annotations, they can make your code more brittle, especially if you annotate with the most initially obvious types. Which is to say, you define some custom classes, and then say that the parameters to and return values from your functions and methods are simply instances of those classes you just defined.

Most Mypy tutorials give you a bunch of examples with str, int, List[int], maybe an Optional[float] or two, and then leave you to your own devices when it comes to defining your own classes; yet, huge amounts of real-world applications are custom classes.

So if you’re new to Mypy, particularly if you’re applying it to a large existing codebase, it’s quite natural to write a little code like this:

1
2
3
4
5
6

from dataclasses import dataclass
@dataclass
class Duck:
    quiet: bool = False
    def quack(self) -> None:
        print("Quack." if self.quiet else "QUACK!")

and then, later, write some code like this:

1
2
3
4

def duck_war(aggressor: Duck, defender: Duck) -> None:
    aggressor.quack()
    defender.quack()
    print("The only winning move is not to play.")

In untyped, pre-Mypy python, in addition to being a poignant message about the futility of escalating violence, duck_war is a very flexible function, regardless of where it’s defined. It can take anything with a quack() method.

But, while the strictness of the Mypy type-check here brings a level of safety — no None accidentally masquerading as a Duck here — it also adds a level of brittleness. Tests which use carefully-constructed fakes will now fail to type check, because duck_war technically insists upon only precisely instances of Duck and nothing else.

A few sub-optimal answers to this question

So when you want something else that has slightly different behavior — when you want a new Duck², so to speak — what do you do?

There are a couple of anti-patterns you might arrive at to work around this as you begin your Mypy journey:

1. add # type: ignore comments to all your tests, or remove their type signatures so they don’t get type checked. This solution throws the baby out with the bath water, as it eliminates any safety that any of these callers experience.
2. add calls to cast(Duck, ...) around any things which you know are “enough like” a Duck for your purposes. This is much more fine-grained and targeted (and can be a great hack for working with libraries who provide type stubs which are too specific) but this also trades off a bit too much safety, since nothing at the point of the cast verifies anything unless you build your own ad-hoc system to do so.
3. subclassing Duck. This can also be expedient, and not too bad if you have to; Mypy removes some of the sharpest edges from Python inheritance, providing some guard rails around overriding methods, but it remains a bad idea for all the usual reasons.

The good answer: typing.Protocol

Mypy has a feature, typing.Protocol , that provides a straightforward way to describe any object that has a quack() method.

You can do this like so:

1
2
3
4
5

from typing import Protocol

class Ducky(Protocol):
    def quack(self) -> None:
        "Quack."

Now, with only a small modification to its signature — while leaving the implementation the same — duck_war can now support anything sufficiently duck-like:

1
2

def duck_war(aggressor: Ducky, defender: Ducky) -> None:
    ...

In addition to making it possible for other code — for example, a unit test — to pass its own implementation of Ducky into duck_war without subclassing or tricking the type checker, this change also improves the safety of duck_war’s implementation itself. Previously, when it took a Duck, it would have been equally valid for duck_war to access the .quiet attribute of Duck as it would have been to access .quack, even though .quiet is ostensibly an internal implementation detail.

Now, we could add an underscore prefix to quiet to make it “private”, but the type checker will still happily let you access it. So a Protocol allows you to clearly reveal your intention about what types you expect your arguments to be.

Why isn’t everything like this?

Unfortunately, typing.Protocol began its life as typing_extensions.Protocol: a custom extended feature of the type system that wasn’t present in Mypy initially, and isn’t in the standard library until Python 3.8. Built-in types like Iterable and Sequence are type-checked as if they’re Protocols by being slightly special within Mypy, but it’s not clear to the casual user how this is happening.

However, other types, like io.TextIO, don’t quite behave this way, and some early-adopter projects for Mypy have types that are either too strict or too permissive because they predated this.

So I really wanted to write this post to highlight the Protocol style of describing types and encourage folks to use it.

In conclusion

The concept I’ve described above is not new in the world of type theory.

The way that typing works in Mypy with most types — builtins, custom classes, and abstract base classes — is known as nominal typing. Nominal as in “based on names”; if the object you have directly references the name of the type it’s being compared to, by being an instance of it, then it matches.

In other words: if it’s named “Duck”, it’s a duck. There are some advantages to nominal typing³, but this brittleness is not very Pythonic!

In contrast, the type of type-checking accomplished by Protocol is known as structural typing.⁴ Whether the caller matches a Protocol depends on the structure of your object — in other words, what methods and attributes it has.

In even other-er words - if it .quack()s like a duck, it is a duck.

If you’re just starting to use Mypy — particularly if you’re building a library that exports types that users are expected to implement — consider using Protocol to describe those types. With Protocol, while you get much-improved safety from type-checking, you don’t lose the wonderful flexibility and easy testability that duck typing has always given you in Python.

There should be one — and preferably only one — obvious way to do it.

Tim Peters, “The Zen of Python”

Moshe wrote a blog post a couple of days ago which neatly constructs a wonderful little coding example from a scene in a movie. And, as we know from the Zen of Python quote, there should only be one obvious way to do something in Python. So my initial reaction to his post was of course to do it differently — to replace an __init__ method with the new @dataclasses.dataclass decorator.

But as I thought about the code example more, I realized there are a number of things beyond just dataclasses that make the difference between “toy”, example-quality Python, and what you’d do in a modern, professional, production codebase today.

So let’s do everything the second, not-obvious way!

There’s more than one way to do it

Larry Wall, “The Other Zen of Python”

Getting started: the __future__ is now

We will want to use type annotations. But, the Guard and his friend are very self-referential, and will have lots of annotations that reference things that come later in the file. So we’ll want to take advantage of a future feature of Python, which is to say, Postponed Evaluation of Annotations. In addition to the benefit of slightly improving our import time, it’ll let us use the nice type annotation syntax without any ugly quoting, even when we need to make forward references.

So, to begin:

1

from __future__ import annotations

Doors: safe sets of constants

Next, let’s tackle the concept of “doors”. We don’t need to gold-plate this with a full blown Door class with instances and methods - doors don’t have any behavior or state in this example, and we don’t need to add it. But, we still wouldn’t want anyone using using this library to mix up a door or accidentally plunge to their doom by accidentally passing "certian death" when they meant certain. So a Door clearly needs a type of its own, which is to say, an Enum:

1
2
3
4
5

from enum import Enum

class Door(Enum):
    certain_death = "certain death"
    castle = "castle"

Questions: describing type interfaces

Next up, what is a “question”? Guards expect a very specific sort of value as their question argument and we if we’re using type annotations, we should specify what it is. We want a Question type that defines arguments for each part of the universe of knowledge that these guards understand. This includes who they are themselves, who the set of both guards are, and what the doors are.

We can specify it like so:

1
2
3
4
5
6
7

from typing import Protocol, Sequence

class Question(Protocol):
    def __call__(
        self, guard: Guard, guards: Sequence[Guard], doors: Sequence[Door]
    ) -> bool:
        ...

The most flexible way to define a type of thing you can call using mypy and typing is to define a Protocol with a __call__ method and nothing else¹. We could also describe this type as Question = Callable[[Guard, Sequence[Guard], Door], bool] instead, but as you may be able to infer, that doesn’t let you easily specify names of arguments, or keyword-only or positional-only arguments, or required default values. So Protocol-with-__call__ it is.

At this point, we also get to consider; does the questioner need the ability to change the collection of doors they’re passed? Probably not; they’re just asking questions, not giving commands. So they should receive an immutable version, which means we need to import Sequence from the typing module and not List, and use that for both guards and doors argument types.

Guards and questions: annotating existing logic with types

Next up, what does Guard look like now? Aside from adding some type annotations — and using our shiny new Door and Question types — it looks substantially similar to Moshe’s version:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

from dataclasses import dataclass

@dataclass
class Guard:
    _truth_teller: bool
    _guards: Sequence[Guard]
    _doors: Sequence[Door]

    def ask(self, question: Question) -> bool:
        answer = question(self, self._guards, self._doors)
        if not self._truth_teller:
            answer = not answer
        return answer

Similarly, the question that we want to ask looks quite similar, with the addition of:

1. type annotations for both the “outer” and the “inner” question, and
2. using Door.castle for our comparison rather than the string "castle"
3. replacing List with Sequence, as discussed above, since the guards in this puzzle also have no power to change their environment, only to answer questions.
4. using the [var] = value syntax for destructuring bind, rather than the more subtle var, = value form

1
2
3
4
5
6
7
8
9

def question(guard: Guard, guards: Sequence[Guard], doors: Sequence[Door]) -> bool:
    [other_guard] = (candidate for candidate in guards if candidate != guard)

    def other_question(
        guard: Guard, guards: Sequence[Guard], doors: Sequence[Door]
    ) -> bool:
        return doors[0] == Door.castle

    return other_guard.ask(other_question)

Eliminating global state: building the guard post

Next up, how shall we initialize this collection of guards? Setting a couple of global variables is never good style, so let’s encapsulate this within a function:

1
2
3
4
5
6
7

from typing import List

def make_guard_post() -> Sequence[Guard]:
    doors = list(Door)
    guards: List[Guard] = []
    guards[:] = [Guard(True, guards, doors), Guard(False, guards, doors)]
    return guards

Defining the main point

And finally, how shall we actually have this execute? First, let’s put this in a function, so that it can be called by things other than running the script directly; for example, if we want to use entry_points to expose this as a script. Then, let's put it in a "__main__" block, and not just execute it at module scope.

Secondly, rather than inspecting the output of each one at a time, let’s use the all function to express that the interesting thing is that all of the guards will answer the question in the affirmative:

1
2
3
4
5
6

def main() -> None:
    print(all(each.ask(question) for each in make_guard_post()))


if __name__ == "__main__":
    main()

Appendix: the full code

To sum up, here’s the full version:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

from __future__ import annotations
from dataclasses import dataclass
from typing import List, Protocol, Sequence
from enum import Enum


class Door(Enum):
    certain_death = "certain death"
    castle = "castle"


class Question(Protocol):
    def __call__(
        self, guard: Guard, guards: Sequence[Guard], doors: Sequence[Door]
    ) -> bool:
        ...


@dataclass
class Guard:
    _truth_teller: bool
    _guards: Sequence[Guard]
    _doors: Sequence[Door]

    def ask(self, question: Question) -> bool:
        answer = question(self, self._guards, self._doors)
        if not self._truth_teller:
            answer = not answer
        return answer


def question(guard: Guard, guards: Sequence[Guard], doors: Sequence[Door]) -> bool:
    [other_guard] = (candidate for candidate in guards if candidate != guard)

    def other_question(
        guard: Guard, guards: Sequence[Guard], doors: Sequence[Door]
    ) -> bool:
        return doors[0] == Door.castle

    return other_guard.ask(other_question)


def make_guard_post() -> Sequence[Guard]:
    doors = list(Door)
    guards: List[Guard] = []
    guards[:] = [Guard(True, guards, doors), Guard(False, guards, doors)]
    return guards


def main() -> None:
    print(all(each.ask(question) for each in make_guard_post()))


if __name__ == "__main__":
    main()

Acknowledgments

I’d like to thank Moshe Zadka for the post that inspired this, as well as Nelson Elhage, Jonathan Lange, Ben Bangert and Alex Gaynor for giving feedback on drafts of this post.

Never send a human to do a machine’s job.

One of the best things about maintaining open source in the modern era is that there are so many wonderful, free tools to let machines take care of the busy-work associated with collaboration, code-hosting, continuous integration, code quality maintenance, and so on.

There are lots of great resources that explain how to automate various things that make maintenance easier.

Here are some things you can configure your Python project to do:

1. Continuous integration, using any one of a number of providers:
   1. GitHub Actions
   2. CircleCI
   3. Azure Pipelines
   4. Appveyor
   5. GitLab CI&CD
   6. Travis CI
2. Separate multiple test jobs with tox
3. Lint your code with flake8
4. Type-Check your code with Mypy
5. Auto-update your dependencies, with one of:
   1. pyup.io
   2. requires.io, or
   3. Dependabot
6. automatically find common security issues with Bandit
7. check the status of your code coverage, with:
   1. Coveralls, or
   2. Codecov
8. Auto-format your code with:
   1. Black for style
   2. autopep8 to fix common errors
   3. isort to keep your imports tidy
9. Help your developers remember to do all of those steps with pre-commit
10. Automatically release your code to PyPI via your CI provider
    1. including automatically building any C code for multiple platforms as a wheel so your users won’t have to
    2. and checking those build artifacts:
       1. to make sure they include all the files they should, with check-manifest
       2. and also that the binary artifacts have the correct dependencies for Linux
       3. and also for macOS
11. Organize your release notes and versioning with towncrier

All of these tools are wonderful.

But... let’s say you¹ maintain a few dozen Python projects. Being a good maintainer, you’ve started splitting up your big monolithic packages into smaller ones, so your utility modules can be commonly shared as widely as possible rather than re-implemented once for each big frameworks. This is great!

However, every one of those numbered list items above is now a task per project that you have to repeat from scratch. So imagine a matrix with all of those down one side and dozens of projects across the top - the full Cartesian product of these little administrative tasks is a tedious and exhausting pile of work.

If you’re lucky enough to start every project close to perfect already, you can skip some of this work, but that partially just front-loads the tedium; plus, projects tend to start quite simple, then gradually escalate in complexity, so it’s helpful to be able to apply these incremental improvements one at a time, as your project gets bigger.

I really wish there were a tool that could take each of these steps and turn them into a quick command-line operation; like, I type pyautomate pypi-upload and the tool notices which CI provider I use, whether I use tox or not, and adds the appropriate configuration entries to both my CI and tox configuration to allow me to do that, possibly prompting me for a secret. Same for pyautomate code-coverage or what have you. All of these automations are fairly straightforward; almost all of the files you need to edit are easily parse-able either as yaml, toml, or ConfigParser² files.

A few years ago, I asked for this to be added to CookieCutter, but I think the task is just too big and complicated to reasonably expect the existing maintainers to ever get around to it.

If you have a bunch of spare time, and really wanted to turbo-charge the Python open source community, eliminating tons of drag on already-over-committed maintainers, such a tool would be amazing.