This blogpost describes the Python development setup that makes the most sense to me (and probably you as well) in 2022 and beyond. By "Development Setup", I mean the local development environment, and not about deploying apps on a server.
The Best Setup, in A Nutshell
For executables/utility tools (e.g. mypy, black), install with system Python + pipx
For project development, install different Python versions via pyenv
For dependency management, use PDM (PDM is installed via pipx)
I'll explain each choice in more details below.
Why Use Both System Python and Pyenv Python?
One unusual thing you probably have noticed is that the setup consists of two ways to install Python: the system Python and Python installed by pyenv. Here, by "system Python", I mean the Python carried with your system, or the Python installed with a system package manager (e.g. apt, brew). Many articles suggested that we should never using system Python, so why still use it? Because we need two independent systems for two different purposes:
For utility tools, reliability matters the most. You want them to always work when you need them. The less things they depend on, the better. System Python is the best choice, because they have the least dependencies. Also, for utility tools, It doesn't really matter which Python it is running on. Therefore even if system Python is behind the latest version, it is usually fine.
For Python interpreters used by projects, flexibility is more important. You may need to upgrade them when a patch version fixed a bug, or using a specific Python version because that's what you have on the server. pyenv fits this purpose well.
pipx
pipx needs no introduction, it has become the de-facto way to install Python executables. But imagine if you install pipx with pyenv-installed Python. This creates a dependency like this:
As you can see, the 3.9.7 (pyenv) --> pyenv part is unnecessary. It also creates trouble when upgrading Python: you either have to reinstall pipx, or keep Python 3.9.7 forever since pipx and all the tools above relies on it.
PDM
Now comes to the controversial part: PDM. Disclaimer: PDM is my personal choice, and it's totally ok if you chose Poetry, Pipenv, or any other tools. Here are the reasons why I use PDM:
⚠️⚠️ UPDATE 2023.5: I should clarify that, PEP 582 is good for developing libraries; for applications, PDM's virtualenv mode should still be preferred.
PDM helped me get away from virtual environments
I used to have 10+ virtual environments, and I get a headache every time I saw the long list. It essentially makes it impossible to ever upgrade my project Python, because I really don't want to setup virtual environments again for every project.
With PDM, I have no virtual environments. Each project directly depends on the pyenv-installed Python. I'm also free to upgrade patch versions (e.g. from 3.9.7 to 3.9.9), which allow me to always use the latest Python release.
PDM is well maintained
PDM is created and maintained by PyPA member Frost Ming. If you don't know what PyPA members do, let's just say they know the best about Python packaging. Frost Ming also maintains Pipenv.
I used to use Poetry, which is also a nice tool. But sometimes Poetry is buggy, especially when upgrading project Python. What's worse is that no one seems to be dealing with those bugs (example). PDM is the opposite. Issues are usually responded within a day or two, including feature requests.
Benefit from being maintained by a PyPA member, PDM is usually the fastest in adopting new proposals in the Python packaging world, which lets its users stay on high productivity.
Overall, I find PDM better than other tools I've used, and it's constantly getting better (5~6 releases per month). I tweeted about my transition from Poetry to PDM, might be interesting to read.
Caveats
You may still need to install a new system Python when the old version gets deprecated. You can use pipx reinstall-all to reinstall existing tools, which is not so bad.
The setup may or may not work on Windows. If you can confirm which case it is, and/or if there's any issues, let me know, I'm happy to put them in the article.
Summary
In the past few years, Python gained a lot of great improvements in the development/packaging ecosystem, but this also created a mess that even the most experienced developers had a hard time picking tools. Luckily, I've found the silver bullet that works for me, and I hope it would also work for you.
我第一次换组的时候也犯过错误,为了能去搞 AR 花了很多时间学图形学,导致没怎么认真看其它组。现在想起来觉得自己真蠢:在 Google 做 AR 没前途不是明摆着的么?到今天为止都没有一款面向消费者的成功 AR 产品,更不要说在这方面投入不多的 Google 了。而且说白了我也不是真的有兴趣,只是觉得这个技术很酷而已。当时犯的错误,现在要用几倍时间来还。
换组心得
下面来聊聊换组。虽然和上一节相关,我还是决定单独拿出来说。很多公司不像 Google 有这么多换组机会,还可能面临老板不放人的情况。因此我不打算写换组的流程,而是把重点放到“选组”上,尽可能让内容通用一些。
PyCascades is a regional PyCon in the Pacific Northwest, celebrating the west coast Python developer and user community. Our organizing team includes members of the Vancouver, Seattle, and Portland Python user groups.
找谁呢?我想起之前看过的那些资料。既然作者有心教人写 proposal,那么应该也会愿意帮忙审阅吧。于是我就给两位作者同时也是圈内大佬 Allison Kaptur 和 Brandon Rhodes 发了邮件。说来也巧,我在调研 Cyberbrain 时仔细阅读了 Allison 写的那篇 A Python Interpreter Written in Python,实现过程中也一直在参考文中提到的 byterun 项目,可以说渊源颇深。邮件发出后好多天也不见回信。我不想傻等,于是就推特私信了 PyCascades 的组织者 Nina。Nina 在一直是 Python 圈的活跃分子,以前也做过关于 debugging 的演讲,我觉得找她还挺合适的。很快有了回信:
This year I'm co-organized for PyCascades, so I currently have my hands full. Fortunately we have a team of volunteers that are providing speaker mentorship for the conference.
Would it be OK if I introduced you to one of our speaker mentors? If yes, just let me know which e-mail address is best for you.
到这步还不能直接录制,组委会要审核 slide 是否符合 Code of Conduct,并做一个 tech check。Tech check 委托了 Next Day Video(一个专业做会议视频的公司)进行,目的是为了确保讲师能够录制出效果足够好的视频。我先通过他们的系统预约,然后顺利进行并通过了 tech check。过程中,Next Day Video 的人还教了我怎么正确使用麦克风——原来我之前一直都摆错了方向导致无法获得最佳音效。果然这就是专业人士吗🤯。然后就是一些邮件+notion文档(1, 2, 3),里面说明了参会和录制的各种要求,详细到视频的 bit rate 和 frame rate 都有规定。里面还推荐了一些录制用的软件。我最后采用的是 Apowersoft Free Online Screen Recorder。CoC review 通过之后,终于可以录制了。
算上 PyCon China 2020 的闪电演讲,这还只是我第二次录视频。我发现,录视频比现场演讲累太多了。可能有人会觉得,现场演讲多紧张啊要面对那么多观众,提前录的话讲坏了大不了重录或者剪辑嘛。但这也正是录视频累人的地方
没有现场观众,你无法得到肾上腺素或者其它什么激素的刺激,而这些刺激能极大缓解疲劳。
你不太会在意现场演讲中的口误、结巴等不完美的地方,因为你在面对别人讲话,而讲话时自然不会在意这些。
提前录制则不同,首先你是自说自话,会对各种错误非常敏感。其次你有重录的机会,导致容易吹毛求疵。
反正我第一次试着完整录,25 分钟,讲完就不行了。那天剩下的时间我几乎没办法干任何事,感觉整个人都被榨干了。自然效果也非常差,后半段可以明显感觉到讲师很疲惫。于是接下来的几天继续录,不算中途就抛弃掉的,大概第三遍的时候我觉得差不多了。我把视频传给了 Next Day Video,几天之后被告知没有问题。接下来就一身轻松等待会议开幕了。
Day 1
主办方在周五安排了一个线上的 social 活动,使用的平台是 SptialChat。这个产品还挺有意思的。进去之后有一个类似剧场的空间,你可以在空间中拖动自己的头像,模拟在物理空间中走动的感觉。当你把头像移到一群人旁边就可以听到他们讲话,并且距离越近听得越清晰。一开始我只是听,后来 Anthony Shaw 过来和我打招呼,我也就顺势混入那一群人瞎聊起来。我说我感觉录视频好累,Anthony 和另一个讲师说他们也发现录视频会让自己要求变高,到最后受不了只能把讲稿完全写下来。
主办方还找了个 DJ 在台上打碟。打碟的大叔全程也不和人交流,感觉很沉醉其中。
Day 2
周六是正式会议,九点半开场。然而我七点就起床看了场球,还输了。郁闷了一阵想起来还要开会,点进去发现 Guido、Brett 等几位大佬已经开始聊天了。随后一直听,并反复在 recorded talk 和 live talk 之间切换,因为 PyCascades 的两个 track 是同时进行的。值得一看的演讲是 Your Name Is Invalid!,大概就是讲处理各种语言文本时各种可能出的问题,比如有的语言中大小写字母不是一一对应的。最后的结论是 "Don't assume anything"。这让我想起以前看过的讲时区处理的视频,想用代码囊括复杂多样的世界,何其难也。
会议之前若干周,主办方在 Slack 上开了几个私密频道给讲师,当作后台使用。会议当天,Next Day Video 的人会在后台联系讲师做 tech check——是的,为了确保万无一失,当天还有 tech check。我的演讲在下午 1:55,大概 1 点做 tech check。工作人员说声音没问题,但让我把后面的百叶窗拉上一点不让太亮了,并且还问能不能把摄像头放低一点。这里要解释一下,我用的是一个台式机连接大显示器,摄像头是放在显示器上面的,这样就造成了一种俯视的感觉。为了解决视角问题,我只能找两个纸巾盒子摞起来,把摄像头放在上面,这样勉强和我的头平齐。我还挺好奇其它人都是怎么做的。
临近演讲时间,我登入 Next Day Video 的讲师专用后台。和主持人对了一次词,包括要怎么称呼和介绍我。这次会议我统一跟人说叫我"laike"了,毕竟称呼不重要。我之前以为开场介绍过后就可以退出,没想到还得一直待在后台。我只能找了一台笔记本打开听自己的演讲。他们强调说视频的播放有一段时间的延迟,但是我听着听着就忘了。临近结尾,我看到 slack 上有消息,是主持人让我马上过来说结束语。也就是说,我需要在视频没结束的时候就去讲,因为有延迟,观众看到的就是视频一结束,我和主持人马上出镜。我手忙脚乱地坐回屏幕前,调整摄像头,讲了一句话才发现自己处于 mute 状态,又赶紧 unmute。这状况频出的直播大概会成为未来的美好回忆吧。