Modularity for Maintenance

Never send a human to do a machine’s job.

One of the best things about maintaining open source in the modern era is that there are so many wonderful, free tools to let machines take care of the busy-work associated with collaboration, code-hosting, continuous integration, code quality maintenance, and so on.

There are lots of great resources that explain how to automate various things that make maintenance easier.

Here are some things you can configure your Python project to do:

1. Continuous integration, using any one of a number of providers:
   1. GitHub Actions
   2. CircleCI
   3. Azure Pipelines
   4. Appveyor
   5. GitLab CI&CD
   6. Travis CI
2. Separate multiple test jobs with tox
3. Lint your code with flake8
4. Type-Check your code with Mypy
5. Auto-update your dependencies, with one of:
   1. pyup.io
   2. requires.io, or
   3. Dependabot
6. automatically find common security issues with Bandit
7. check the status of your code coverage, with:
   1. Coveralls, or
   2. Codecov
8. Auto-format your code with:
   1. Black for style
   2. autopep8 to fix common errors
   3. isort to keep your imports tidy
9. Help your developers remember to do all of those steps with pre-commit
10. Automatically release your code to PyPI via your CI provider
    1. including automatically building any C code for multiple platforms as a wheel so your users won’t have to
    2. and checking those build artifacts:
       1. to make sure they include all the files they should, with check-manifest
       2. and also that the binary artifacts have the correct dependencies for Linux
       3. and also for macOS
11. Organize your release notes and versioning with towncrier

All of these tools are wonderful.

But... let’s say you¹ maintain a few dozen Python projects. Being a good maintainer, you’ve started splitting up your big monolithic packages into smaller ones, so your utility modules can be commonly shared as widely as possible rather than re-implemented once for each big frameworks. This is great!

However, every one of those numbered list items above is now a task per project that you have to repeat from scratch. So imagine a matrix with all of those down one side and dozens of projects across the top - the full Cartesian product of these little administrative tasks is a tedious and exhausting pile of work.

If you’re lucky enough to start every project close to perfect already, you can skip some of this work, but that partially just front-loads the tedium; plus, projects tend to start quite simple, then gradually escalate in complexity, so it’s helpful to be able to apply these incremental improvements one at a time, as your project gets bigger.

I really wish there were a tool that could take each of these steps and turn them into a quick command-line operation; like, I type pyautomate pypi-upload and the tool notices which CI provider I use, whether I use tox or not, and adds the appropriate configuration entries to both my CI and tox configuration to allow me to do that, possibly prompting me for a secret. Same for pyautomate code-coverage or what have you. All of these automations are fairly straightforward; almost all of the files you need to edit are easily parse-able either as yaml, toml, or ConfigParser² files.

A few years ago, I asked for this to be added to CookieCutter, but I think the task is just too big and complicated to reasonably expect the existing maintainers to ever get around to it.

If you have a bunch of spare time, and really wanted to turbo-charge the Python open source community, eliminating tons of drag on already-over-committed maintainers, such a tool would be amazing.