Migrating a project to Poetry

Poetry is a tool solving the problem of Python packaging. It was started back in February 2018 by Sébastien Eustace (also the author of pendulum). It has a beautiful website and a ambitious headline:

Python packaging and dependency management made easy

It has been on my radar for a while, but I never gave it a proper go. I was happily using pip-tools, which was solving my main use case, while being a lot more lightweight and meant I could keep working with pip since its output is a good old requirements.txt. I heard about it a few times online, often next to Pipenv, but recently it looks like Poetry got a bit more traction.

Having a bit of time on my hands, a few weeks ago I decided to take a proper look at it and maybe migrate one of my projects, Deezer Python.

Starting point

Before the migration, here is how the package was managed:

  • Project metadata were in setup.cfg, using setuptools declarative config.
  • Development (and documentation) dependencies managed by pip-tools.
  • Documentation hosted on Read the Docs (RTD).
  • Releases automated with Python Semantic Release (PSR) on Github Actions.
  • Development tools configured in setup.cfg (black, isort, pyupgrade, flake8).
  • Using tox to help local testing.
  • Using Github Actions for CI.

These are a couple of features that were impacted by migrating to Poetry, and I think it’s worth mentioning them for context. Since Poetry uses pyproject.toml, I was also hoping to move all my tools config from setup.cfg to that file.

Migration

Installing Poetry

The first step is to get the CLI. I initially installed it via Homebrew, but later realised that Poetry was setting some default values based on the Python version its installation uses. As the Homebrew Python can be updated without notice, I realised it was not the best option here, so I later reinstalled it via pipx using a Python installation managed by pyenv that wouldn’t be wiped without my knowledge:

pipx install \
 --python ~/.pyenv/versions/3.8.6/bin/python \
 poetry

Project metadata

The first step was to migrate the project metadata from setup.cfg to pyproject.toml. Poetry comes with a handy interactive command poetry init which will create a minimal pyproject.toml for you. I already noticed a few pleasant surprises:

  • The CLI was very nice to interact with
  • The author and author_email from setup.cfg were merge into an array of authors, each with the format Full Name <[email protected]>.

I then went on to convert more settings into the new format manually, and the process was quite painless. Many settings have the same name and values, and when they are different, it’s mainly to simplify things. I guess it’s something a library like setuptools cannot easily afford to do due to backwards compatibility, but that a new opt-in tool like Poetry can.

Dependencies

Adding dependencies and development dependencies was pretty simple, I just needed to run poetry add [-D] ... with the list of packages at the end.

Extra Dependencies

This package had an “extra require” dependency. There is a good example in the documentation, these dependencies needs to be specified in the same section as normal dependencies, but as optional:

[tool.poetry.dependencies]
...
tornado = {version = "^6.0.4", optional = true}

And a dedicated pyproject.toml section maps the extras to an array of optional dependencies:

[tool.poetry.extras]
tornado = ["tornado"]

I got confused initially because I thought it was done via the poetry add ... -E ... command. However, it does something different, it’s for the extra of the dependency, not the extra of my own library the dependency should fall under.

For example Django has 2 possible extras at the moment, so one would run the following commands to use them:

poetry add Django -E argon2

It means “install Django with the argon2 extra”. I thought it meant “install Django and put it under the argon2 extra”.

Docs dependencies

The dependencies to build the docs were specified in a requirements.txt in the docs/ folder and RTD was configured to pick this up. I initially thought that I wouldn’t be able to remove that file, but it turns out it’s possible to make it work.

Thanks to PEP 517, which Poetry is compliant with, you can do pip install . in a Poetry package. This has been in pip since 19.0, and pip running on RTD is newer than this. However, this method wouldn’t install your development dependencies, so your docs dependencies cannot be specified as such. It works if you specify a docs extra, though:

# pyproject.toml
[tool.poetry.dependencies]
...
myst-parser = {version = "^0.12", optional = true}
sphinx = {version = "^3", optional = true}
sphinx-autobuild = {version = "^2020.9.1", optional = true}
sphinx-rtd-theme = {version = "^0.5", optional = true}

[tool.poetry.extras]
...
docs = [
    "myst-parser",
    "sphinx",
    "sphinx-autobuild",
    "sphinx-rtd-theme",
]
# readthedocs.yml
version: 2
python:
  install:
    - method: pip
      path: .
      extra_requirements:
        - docs

The downside of this, is that the extra is part of your package, so not ideal.

Releases

I recently moved the automation of releases to Python Semantic Release which worked well for me, and this would have been a blocker if it wouldn’t work. These are the pieces I needed:

  • Move its config to come from pyproject.toml.
  • Package version is specified in pyproject.toml as well as in __init__.py.
  • Update build command to use Poetry instead of setuptools.

Here is the PSR config in pyproject.toml to achieve that:

[tool.semantic_release]
version_variable = [
    "deezer/__init__.py:__version__",
    "pyproject.toml:version"
]
build_command = "pip install poetry && poetry build"

I was expecting to have to change more, but it all worked out of the box with just that.

Linting and code formatting

All the tools I use for linting and code formatting were configured via setup.cfg and ideally I’d like to replace it by pyproject.toml. It was possible for almost everything, except for flake8 which has an open issue for it.

I decided to move as much things as I could to pyproject.toml, and move flake8 config to .flake8.

With all the above, I was able to remove the setup.cfg as well as all the Pip Tools files for dependencies.

Tox

Poetry works nicely with Tox, I followed the section in their FAQ, and here is a overview of the changes:

[tox]
isolated_build = true
envlist = py36,py37,py38,py39,pypy3,docs,lint,bandit

[testenv]
whitelist_externals = poetry
commands =
    poetry install
    poetry run pytest
...

I replaced the deps section in each testenv by a poetry install into the list of commands to run, and prefixed all commands to be run in the isolated environment by poetry run.

Github Actions

Poetry isn’t installed out of the box on Github Actions, one could either install it with a simple run step or use a dedicated action for it. I’ve opted for the dedicated action, thinking that Dependabot could keep it up to date for me.

The rest of the changes are pretty simple, it’s a matter or replacing pip install by poetry install -E ... and prefixing all commands by poetry run. My docs were tested and I was changing directory with cd, I took this opportunity to instead use working-directory key to the Github action step.

Verdict

Did Poetry deliver on its ambitious tagline? I think so, I was really impressed by the developer experience of Poetry, its CLI is really nice, and I hit little issues on the way. Overall the migration was not too difficult, you can check the pull request on Github. I feel like there are quite a few features I just scratched the surface (like multi-environments). I’m going to wait a bit to see how this works in the longer run, but I think I’ll migrate my other projects soon.