Migrating a project to Poetry

Poetry is a tool solving the problem of Python packaging. It was started back in February 2018 by Sébastien Eustace (also the author of pendulum). It has a beautiful website and an ambitious headline:

Python packaging and dependency management made easy

It has been on my radar for a while, but I never gave it a proper go. I was happily using pip-tools, which was solving my main use case, while being a lot more lightweight and meant I could keep working with pip since its output is a good old requirements.txt. I heard about it a few times online, often next to Pipenv, but recently it looks like Poetry got a bit more traction.

Having a bit of time on my hands, a few weeks ago I decided to take a proper look at it and maybe migrate one of my projects, Deezer Python.

Starting point

Before the migration, here is how the package was managed:

• Project metadata were in setup.cfg, using setuptools declarative config.
• Development (and documentation) dependencies managed by pip-tools.
• Documentation hosted on Read the Docs (RTD).
• Releases automated with Python Semantic Release (PSR) on Github Actions.
• Development tools configured in setup.cfg (black, isort, pyupgrade, flake8).
• Using tox to help local testing.
• Using Github Actions for CI.

These are a couple of features that were impacted by migrating to Poetry, and I think it’s worth mentioning them for context. Since Poetry uses pyproject.toml, I was also hoping to move all my tools config from setup.cfg to that file.

Migration

Update: I recently came across the dephell tool which seems to automate some conversions check out the follow-up post.

Installing Poetry

The first step is to get the CLI. I initially installed it via Homebrew, but later realised that Poetry was setting some default values based on the Python version its installation uses. As the Homebrew Python can be updated without notice, I realised it was not the best option here, so I later reinstalled it via pipx using a Python installation managed by pyenv that wouldn’t be wiped without my knowledge:

pipx install \
 --python ~/.pyenv/versions/3.8.6/bin/python \
 poetry

Project metadata

The first step was to migrate the project metadata from setup.cfg to pyproject.toml. Poetry comes with a handy interactive command poetry init which will create a minimal pyproject.toml for you. I already noticed a few pleasant surprises:

• The CLI was very nice to interact with
• The author and author_email from setup.cfg were merged into an array of authors, each with the format Full Name <[email protected]>.

I then went on to convert more settings into the new format manually, and the process was quite painless. Many settings have the same name and values, and when they are different, it’s mainly to simplify things. I guess it’s something a library like setuptools cannot easily afford to do due to backwards compatibility, but that a new opt-in tool like Poetry can.

Dependencies

Adding dependencies and development dependencies was pretty simple, I just needed to run poetry add [-D] ... with the list of packages at the end.

In this process, I discovered that one of the development dependencies, pyupgrade is not compatible with Python 3.6.0: Poetry would not let me set my own Python to ^3.6. I initially changed my minimum version to ^3.6.1 as a quick fix, but I realised later that it impacted my package’s trove classifiers, the ones generated by Poetry: my package wasn’t listed as Python 3.6 compatible. Since this is just for a development dependency, there is a better way! The fix is to specify the dependency conditional to the Python version:

[tool.poetry.dependencies]
python = "^3.6"
...

[tool.poetry.dev-dependencies]
...
pyupgrade = { version = "^2.7.3", python = "^3.6.1" }

With this, the minimum Python version of my package and its classifiers are correct.

Extra Dependencies

This package had an “extra require” dependency. There is a good example in the documentation, these dependencies need to be specified in the same section as normal dependencies, but as optional:

[tool.poetry.dependencies]
...
tornado = {version = "^6.0.4", optional = true}

And a dedicated pyproject.toml section maps the extras to an array of optional dependencies:

[tool.poetry.extras]
tornado = ["tornado"]

I got confused initially because I thought it was done via the poetry add ... -E ... command. However, it does something different, it’s for the extras of the dependency, not the extras of my own library the dependency should fall under.

For example Django has 2 possible extras at the moment, so one would run the following command to use them:

poetry add Django -E argon2

It means “install Django with the argon2 extra”. I thought it meant “install Django and put it under the argon2 extra”.

Docs dependencies

The dependencies to build the docs were specified in a requirements.txt in the docs/ folder and RTD was configured to pick this up. I initially thought that I wouldn’t be able to remove that file, but it turns out it’s possible to make it work.

Thanks to PEP 517, which Poetry is compliant with, you can do pip install . in a Poetry package. This has been in pip since 19.0, and pip running on RTD is newer than this. However, this method wouldn’t install your development dependencies, so your docs dependencies cannot be specified as such. It works if you specify a docs extra, though:

# pyproject.toml
[tool.poetry.dependencies]
...
myst-parser = {version = "^0.12", optional = true}
sphinx = {version = "^3", optional = true}
sphinx-autobuild = {version = "^2020.9.1", optional = true}
sphinx-rtd-theme = {version = "^0.5", optional = true}

[tool.poetry.extras]
...
docs = [
    "myst-parser",
    "sphinx",
    "sphinx-autobuild",
    "sphinx-rtd-theme",
]

# readthedocs.yml
version: 2
python:
  install:
    - method: pip
      path: .
      extra_requirements:
        - docs

The downside of this, is that the extra is part of your package, so not ideal.

Releases

I recently moved the automation of releases to Python Semantic Release which worked well for me, and this would have been a blocker if it hadn’t worked. These are the pieces I needed:

• Move its config to come from pyproject.toml.
• Package version is specified in pyproject.toml as well as in __init__.py.
• Update build command to use Poetry instead of setuptools.

Here is the PSR config in pyproject.toml to achieve that:

[tool.semantic_release]
version_variable = [
    "deezer/__init__.py:__version__",
    "pyproject.toml:version"
]
build_command = "pip install poetry && poetry build"

I was expecting to have to change more, but it all worked out of the box with just that.

Linting and code formatting

All the tools I use for linting and code formatting were configured via setup.cfg and ideally I’d like to replace it by pyproject.toml. It was possible for almost everything, except for flake8 which has an open issue for it.

I decided to move as many things as I could to pyproject.toml, and move flake8 config to .flake8.

With all the above, I was able to remove the setup.cfg as well as all the pip-tools files for dependencies.

Code Coverage

Pytest had an unexpected config section for pyproject.toml, I didn’t pay much attention to it initially and put its config under [tool.pytest], while the section should actually be [tool.pytest.ini_options].

The consequence of this was that my config was ignored and I didn’t run the tests with coverage enabled, which silenced another error in the coverage section tool.coverage.run, where source should be an array:

[tool.pytest.ini_options]addopts = "-v -Wdefault --cov=deezer"

[tool.coverage.run]
branch = true
source = ["deezer"]

With this, pytest was running with coverage, but for some reason, codecov wasn’t picking up the report properly. I noticed that it used to find an xml file when it was working, so I edited the command on CI to generate it with --cov-report=xml:

poetry run pytest --cov-report=xml

With these changes I eventually got my code coverage reporting back, but it’s something I missed in the original migration. Not directly related to Poetry, but interesting that Pytest decided to go with this section name.

Tox

Poetry works nicely with Tox, I followed the section in their FAQ, and here is an overview of the changes:

[tox]
isolated_build = trueenvlist = py36,py37,py38,py39,pypy3,docs,lint,bandit

[testenv]
whitelist_externals = poetrycommands =
    poetry install    poetry run pytest...

I replaced the deps section in each testenv by a poetry install into the list of commands to run, and prefixed all commands to be run in the isolated environment by poetry run.

Github Actions

Poetry isn’t installed out of the box on Github Actions, one could either install it with a simple run step or use a dedicated action for it. I’ve opted for the dedicated action, thinking that Dependabot could keep it up to date for me.

The rest of the changes are pretty simple, it’s a matter of replacing pip install by poetry install -E ... and prefixing all commands by poetry run. My docs were tested and I was changing directory with cd, I took this opportunity to instead use working-directory key to the Github action step.

Verdict

Did Poetry deliver on its ambitious tagline? I think so, I was really impressed by the developer experience of Poetry, its CLI is really nice, and I hit few issues on the way. Overall the migration was not too difficult, you can check the pull request on Github. I feel like there are quite a few features where I only scratched the surface (like multi-environments). I’m going to wait a bit to see how this works in the longer run, but I think I’ll migrate my other projects soon.