This blogpost describes the Python development setup that makes the most sense to me (and probably you as well) in 2022 and beyond. By "Development Setup", I mean the local development environment, and not about deploying apps on a server.
The best setup, in a nutshell
- For executables/utility tools (e.g. mypy, black), install with system Python + pipx
- For project development, install different Python versions with pyenv
- For dependency management, use PDM
I'll explain each choice in more details below.
Why have two Python systems?
One unusual thing you probably have noticed is that the setup consists of two ways to install Python: the system Python and Python installed by pyenv. Here, by "system Python", I mean the Python carried with your system, or the Python installed with a system package manager (e.g.
brew). Many articles suggested that we should never using system Python, so why still use it? Because we need two independent systems for two different purposes:
- For utility tools, reliability matters the most. You want them to always work when you need them. The less things they depend on, the better. System Python is the best choice, because they have the least dependencies. Also, for utility tools, It doesn't really matter which Python it is running on. Therefore even if system Python is behind the latest version, it is usually fine.
- For Python interpreters used by projects, flexibility is more important. You may need to upgrade them when a patch version fixed a bug, or using a specific Python version because that's what you have on the server. pyenv fits this purpose well.
pipx needs no introduction, it has become the de-facto way to install Python executables. But imagine if you install pipx with pyenv-installed Python. This creates a dependency like this:
mypy --> pipx --> 3.9.7 (pyenv) --> pyenv --> system Python
As you can see, the
3.9.7 (pyenv) --> pyenv part is unnecessary. It also creates trouble when upgrading Python: you either have to reinstall pipx, or keep Python 3.9.7 forever since pipx and all the tools above relies on it.
Now comes to the controversial part: PDM. Disclaimer: PDM is my personal choice, and it's totally ok if you chose Poetry, Pipenv, or any other tools. Here are the reasons why I use PDM:
PDM helped me get away from virtual environments
I used to have 10+ virtual environments, and I get a headache every time I saw the long list. It essentially makes it impossible to ever upgrade my project Python, because I really don't want to setup virtual environments again for every project.
With PDM, I have no virtual environments. Each project directly depends on the pyenv-installed Python. I'm also free to upgrade patch versions (e.g. from 3.9.7 to 3.9.9), which allow me to always use the latest Python release.
See also You don't really need a virtualenv.
PDM is well maintained
PDM is created and maintained by PyPA member Frost Ming. If you don't know what PyPA members do, let's just say they know the best about Python packaging. Frost Ming also maintains Pipenv.
I used to use Poetry, which is also a nice tool. But sometimes Poetry is buggy, especially when upgrading project Python. What's worse is that no one seems to be dealing with those bugs (example). PDM is the opposite. Issues are usually responded within a day or two, including feature requests.
Benefit from being maintained by a PyPA member, PDM is usually the fastest in adopting new proposals in the Python packaging world, which lets its users stay on high productivity.
You may still need to install a new system Python when the old version gets deprecated. You can use
pipx reinstall-all to reinstall existing tools, which is not so bad.
The setup may or may not work on Windows. If you can confirm which case it is, and/or if there's any issues, let me know, I'm happy to put them in the article.
In the past few years, Python gained a lot of great improvements in the development/packaging ecosystem, but this also created a mess that even the most experienced developers had a hard time picking tools. Luckily, I've found the silver bullet that works for me, and I hope it would also work for you.comments powered by Disqus