Bruno Alla's Blog

Introducing django-remake-migrations

Mon, 03 Mar 2025 00:00:00 GMT

Django migrations framework is a great tool to evolve your database schema over time allowing you to make pretty much any change to your data model (create/drop tables, add, change or remove columns, create or drop index) as well as writing data migrations when you need to move data around. Each time you make a change to your model, Django creates a new migration file with the diff. Sometimes it changes the database, but sometimes it's only changing the model state.

However, it can easily become an append-only system, and on a project of a decent size (medium to large), historical migrations can start to slow down your development cycle, for example if you run all migrations on CI, or for deployment preview environment linked to each pull request. In these 2 cases, it's quite common to run all migrations from scratch, and once the project went through enough model changes, it can easily take several minutes.

The built-in way

Django provides a built-in squashmigrations command to help reduce the amount of migrations files, it works on a single app and takes a range of migrations to merge together into a single file, trying to optimize the operations along the way. However, Django - having the user base it has - needs to be absolutely certain and correct about which optimizations to perform, and can only do so much.

A bigger problem, in my opinion, is the limitation to work a single app only. Projects tend to make use of Django apps to modularize the codebase, but without strict guardrails, it's very easy to introduce circular inter-apps dependencies and after enough time, it becomes practically impossible to squash migrations. You want to squash migrations in app A, but they depend on app B. App B depends on app C which in turn depends on app A. If your project has a dozen apps, you quickly get many cycles like this.

How it works

The built-in squash migration command works by looking at all the operations in the migration files, putting them all together in a single file. But what happens when the squash migrations are deployed? How does Django know to not run them again and create a table that in fact already exists? It's marking the new migration file as replacing the original migrations. Here is an example, consider the following 2 migrations files:

# 0001_initial.py
class Migration(migrations.Migration):
    initial = True

    operations = [
        migrations.CreateModel(
            name="Book",
            fields=[
                (
                    "id",
                    models.BigAutoField(
                        auto_created=True,
                        primary_key=True,
                        serialize=False,
                        verbose_name="ID",
                    ),
                ),
            ],
        ),
    ]

# 0002_add_title.py
class Migration(migrations.Migration):
    dependencies = [
        ("library", "0001_initial"),
    ]

    operations = [
        migrations.AddField(
            model_name="book",
            name="title",
            field=models.CharField(
                max_length=255,
            ),
        ),
    ]

If we squash them, we'd probably get something along these lines:

# 0001_squashed_initial
class Migration(migrations.Migration):
    initial = True

    replaces = [
        ("library", "0001_initial"),
        ("library", "0002_add_title"),
    ]

    operations = [
        migrations.CreateModel(
            name="Book",
            fields=[
                (
                    "id",
                    models.BigAutoField(
                        auto_created=True,
                        primary_key=True,
                        serialize=False,
                        verbose_name="ID",
                    ),
                ),
            ],
        ),
        migrations.AddField(
            model_name="book",
            name="title",
            field=models.CharField(
                max_length=255,
            ),
        ),
    ]

Notice how the 2 operations are put together in the same file, and the squashed migration file is marked as replacement of the previous 2 using the replaces attribute. This is what tells Django not actually execute the operations in the squashed migration file, unless the migrations it replaces haven't run.

Introducing django-remake-migrations

Building on this feature, django-remake-migrations bring the power of squashmigrations to all the first party apps in your project. It provides a new command called remakemigrations that:

Delete all the migration files
Call Django's makemigrations command to generate the minimum number of files and operations from scratch
Mark all these new migrations as squashed using the replaces attribute, to prevent them from executing on system which are fully migrated.

However, this comes with a pretty big caveat: the replaces field is pretty much guaranteed to be wrong in terms of which migrations are replaced by which, but if all your deployments are fully migrated, it doesn't matter much in practice.

On a medium-sized project, it can be tricky to time such deployment and avoid conflicting with another teammate who might add a migration somewhere. For that, I highly recommend using django-linear-migrations as it will help you flag such conflict in git, hence preventing unknowingly concurrent merges. The library provides an integration hook to regenerate max-migrations files at the end. There are a few options if you need to include some database extensions or if you want to prioritize certain apps to generate migrations for.

Conclusion

This probably won't work on very large codebases with 100's of devs, or for projects that don't fully control all the deployments. However, I believe it works quite well for a vast majority of Django codebases, where a few dozen people work on the project at once.

Project-level shareable live templates in PyCharm (or other IntelliJ IDEs)

Fri, 21 Feb 2025 00:00:00 GMT

Most code editors have the ability to scaffold boilerplate with small pieces of templates: type a keyword and get some code generated for you, with the cursor moved to where you need to fill in the blank. While LLMs are getting better at suggesting code for us, these snippets are usually more precise, faster and easier to direct/nudge than an LLM, and serve a different purpose than what an LLM does.

IntelliJ calls them live templates, VS Code calls them snippets. Emmet works as a plugin for all editors in this category too.

The missing feature

One feature that VS Code has is the ability to define project level snippets. While working on a big codebase with a team, common patterns tend to emerge and having them encoded in custom snippets help to nudge everyone in following them.

However, not everyone uses VS Code. I use PyCharm and I figured I was missing out on these project level snippets. I had some custom live templates but sharing them with my fellow PyCharm users wasn't as straightforward.

Finding a solution

After a bit of research, I discovered where my custom live templates are stored by my editor. Turns out that, on macOS, they are stored as xml in ~/Library/Application Support/JetBrains/PyCharm2024.3/templates/ (with PyCharm2024.3 whatever edit and version you're on). I wondered if I could make one a symbolic link to a folder in my repo and sure enough it worked.

From the editor UI, under Settings > Editor > Live templates
Add a new Template Group, e.g. "MyGroup"
Add a live template in it
Go to the templates directory from earlier, a new XML file with the same name should appear
Copy that file into your repo, in a memorable location. I went with .pycharm/ folder at the root of my repo.

Create a symlink from where PyCharm looks pointing at your repo (assuming it's at /path/to/repo/):

ln -s \
 /path/to/repo/.pycharm/MyGroup.xml \
 ~/Library/Application\ Support/JetBrains/PyCharm2024.3/templates/MyGroup.xml

After that, I had to restart my editor for the new file to be picked up. I did a few updates and the symlink was correctly followed on writes: the live templates updated in the UI appeared in the file stored in the repo.

Install script

That worked well, but the instructions to install them were a bit fiddly, mainly because it requires to pass some absolute paths which are dynamic and prone to change in future versions. I figured I could write a simple bash script to do create the symlink in the right place and make a simple installation.

With a bit of help from the Codium plugin in my editor, I ended with a good enough script that looks like this:

#!/usr/bin/env bash

# Resolve path to the Legl.xml file in this directory
PARENT_DIR="$(dirname "$0")"
FULL_PATH="$(pwd)/$PARENT_DIR/MyGroup.xml"

# Get folder with highest number matching
# ~/Library/Application\ Support/JetBrains/PyCharm20*
LARGEST_VERSION_DIR=$(ls -d ~/Library/Application\ Support/JetBrains/PyCharm20* | sort -V | tail -1)

ln -s \
  "$FULL_PATH" \
  "$LARGEST_VERSION_DIR/templates/MyGroup.xml"

it's probably not catering for all use cases but as I said it's good enough. Now my teammates can run that install script with .pycharm/install.sh and the live templates are synced back and forth.

Migrating from Chakra UI to Tailwind CSS

Fri, 24 Jan 2025 00:00:00 GMT

I wrote before how I migrated to ChakraUI using ChatGPT to guide me and how that sped up the task. Less than a year later, I migrated again my style system but this time it was even more impressive, and even faster.

The migration

Having heard about Cursor a lot lately, and having seen it in action with some colleagues at work, I decided to give it a proper go and was really impressed. I opened the "composer" tool in "agent" mode, opened my codebase and asked to "Replace ChakraUI by Tailwind CSS with the minimum amount of visual differences possible".

The agent went off and started looking at the structure of the repo, and my dependencies, detected that I was running Gatsby and suggested a plan. Then it suggested running npm install ... command which sadly failed with permissions issues. I ran them myself in another terminal, and told it that to proceed. I sat and watch him make modifications to the codebase, going through my components and migrating them one by one. It then it claimed to have completed it and ended by uninstalling the old dependency (I had to run this manually again due to permissions issues).

I this point I tried to start the site, and it crashed as the migration was in fact not fully completed. I asked it to finish migrating the components and it finally did it. It then asked whether it should proceed with other folders which I instructed it to do it, and eventually reached a point where Chakra UI was gone.

To my amazement, the site started and looked pretty close to what I had before. This whole process took maybe 15 or 20 mins and I didn't type or supervised it very much. I was really impressed. I then only had to do some final tweaks to get the styles closer to what I wanted. I never used Tailwind CSS before so it was a massive speed up.

Why migrate

As time of writing, this blog is built with GatsbyJS. Unfortunately, the project has been dormant for quite a while now, pretty much since Netlify bought them. At this point, I consider the project dead and while there is no urgency to migrate, some of the tolling starts to fall apart, and with the velocity of Javascript development, I'm concerned that might not be able to migrate in a couple of months time if I don't do it now, as the migration guide might go away one day.

An interesting framework that I might want to try is Astro, but from what I can tell, there is not real integration with Chakra UI (not mentioning that Chakra UI released a new version which doesn't support GatsbyJS AFAIK).

All this made me realise that my design system is too much coupled with the framework I use (React/Gatsby), and that I would benefit to switch to something a bit more framework-agnostic. I never tried Tailwind before, as it wasn't appealing to me, I still don't get the whole utility classes, but I'm giving it a try. Perhaps it's because I'm not a CSS developer or because I'm not working on a big team...

Anyway, I'm hoping that Tailwind, with its huge community will have support in most places, and won't tie me to any framework.

Next steps

I'll do some more iterations to limit my Tailwind usage to reusable components only, which I think should help if I decide to move off something React-based. That experiment taught me that while doing these kind of migrations used to be a really painful, it will be getting easier and easier, these AI tools are great application for them.

Testing your Python logging like a boss

Fri, 13 Dec 2024 00:00:00 GMT

The problem

I recently wanted to add a test to some piece of my code to make sure I would be properly notified in case of certain problem occurred. The system in question uses Sentry, so I was planning to use logger.error() from the standard library, which would create an issue for me.

I wanted to make sure I had the relevant details in my message, hence why I wanted a test for it.

The naive solution

The initial solution I reached for was to mock the logger and assert calls matched what I needed:

# script.py
import logging

logger = logging.getLogger(__name__)

def run(group):
    if missing_fields := group.has_missing_fields():
        logger.error(
            "Missing data for group %s: %r",
            group,
            missing_fields
        )
        return

    # ... Proceed

And here is how my test looked like:

# test_script.py
def test_error_logging(mocker):
    group_with_missing_data = Group()
    logger = mocker.patch("script.logger", autospec=True)

    run(group_with_missing_data)

    logger.exception.assert_called_once_with(
        "Missing data for group %s: %r",
        group_with_missing_data,
        ["name"],
    )

The problem was that the test was not very meaningful:

it was very close from the implementation, almost mirroring it, and felt a bit like writing "2+2 == 2+2" instead of "2+2 == 4".
it didn't convey how the final formatted message would look like, I wanted to make sure that the structure were readable

Another alternative was to format the message using an f-string, but that's a discouraged practice, mainly because it's eagerly formatting the string. I also noticed that monitoring tools like Sentry tend have a harder time at grouping multiple instances of the same error when the message is formatted as f-string, as opposed to passing parameters separately.

The better solution

I wasn't happy with my solution, and kept thinking that there's got to be a better way of doing this. I had a bit of spare time to explore so I researched a bit and found a much better solution: pytest provides a caplog fixture. This is exactly the right tool for the job here, which enabled me to change my test to:

# test_script.py
def test_error_logging(caplog):
    caplog.set_level(logging.ERROR)
    group_with_missing_data = Group()

    run(group_with_missing_data)

    assert len(caplog.records) == 1
    assert caplog.records[0].levelname == "ERROR"
    assert caplog.records[0].message == (
        "Missing data for group <Group >: 'name'"
    )

This enables to set the level of logging you're interested in capturing, and make assertions on each log record. Each log record has the final formatted message, the unformatted messages, its parameters, the log level as well as a number of other things.

Without pytest

The standard library TestCase class from the unittests module has an assertLogs method which offers a similar functionality.

Closing words

It was a nice discovery and I felt more confident that the alert from my monitoring will be useful when something fails. It's something I'll definitely use again next time I need it.

Manage deprecations with Python warnings in a Django project

Fri, 22 Nov 2024 00:00:00 GMT

Python has a warnings module in its standard library that can be very helpful tool to manage deprecations. It's commonly used by libraries but can also be useful to manage internal changes on big projects with lots of concurrent changes.

The library use case

Libraries with lots of users sometimes want to notify their users of upcoming changes that they plan break in the future. On the receiving end, users can control which action to take, and this action may be different depending on the context.

For instance, Django defines (at time of writing) a few warnings RemovedInDjango60Warning, RemovedInDjango61Warning, aliased to respectively RemovedInNextVersionWarning and RemovedAfterNextVersionWarning. While it may be useful to see these warnings in local development or while running tests on CI, it could be noisy to have them on production, potentially distracting us from other important signals.

Warnings can be filtered in various ways:

with the -W command line argument when running Python (python -W manage.py ...)
with the PYTHONWARNINGS environment variable
programmatically, in your Python code with warnings.filterwarnings(...)
in pytest, using the -W option or via the filterwarnings config.

I usually try to enable all warnings during my CI runs, and not show them anywhere else. If there is too much noise in my CI, then it's a sign that I should fix the warnings. If you need some more advanced filtering, the filtering syntax is quite flexible, the pytest documentation does a good job at giving a few examples of filters.

Using warnings internally

Warnings can also be useful to manage deprecations within the bound of an application, without external downstream consumers of the code, but with lots of developers. In big project, it's quite common to have a shared internal library of utilities to make things easier to do, or just have service layers that other parts of the system use. After a while, some of these parts can end up being used 100s or 1000s of times.

When we inevitably reach the limit of a given system and want to refactor it towards a better approach, it can take time to transition. In some cases, data might need to be migrated over in the middle, and should to be released in stages to avoid downtime. While the 2 or 3 stages are implemented, the rest of team keeps moving at full speed, and may keep using the old pattern instead of using the new way of doing things.

Emitting a custom warning is a good way to manage these gradual deprecations. It's much easier to control what happens on CI that it is on production, and you can opt to treat your warning as an error in this context, hence breaking the tests and signaling others towards the new way of doing things.

Example

Imagine you have a is_user_allowed function in an internal library that you want to deprecate in favour of a new_is_user_allowed. This function may be used in lots of places, to do higher level operations. You could add a call to emit a warning in your deprecated function:

# consumer.py
from lib import is_user_allowed

def do_high_level_stuff(user):
    is_user_allowed(user)
    ...

# lib.py
import warnings

def is_user_allowed(user):
    warnings.warn(
        "is_user_allowed is deprecated",
        category=DeprecationWarning,
        stacklevel=2,
    )
    return new_is_user_allowed(user)

It's a good idea to pass a stacklevel of at least 2, otherwise the warning will appear to come from lib.py. By passing a stacklevel of 2, the warning will appear to come from consumer.py, hence making it much easier to locate usages.

You can configure pytest to treat this warning as error, except in existing places where it's already used:

# pyproject.toml
[tool.pytest.ini_options]
filterwarnings = [
    "once",
    "error:is_user_allowed is deprecated:DeprecationWarning",
    "once:is_user_allowed is deprecated:DeprecationWarning:consumer",
    "...",
]

If other teammates working on other branches start using it in new places, their test will start failing, preventing them to introduce new usages of the deprecated function. This first step can be released while you gradually fix all the existing usages, removing them from the pytest's filterwarnings list as you go.

You may want to go one step further and log usages on production for a little while. Warnings aren't integrated with the standard logging module by default, but it's not very hard to setup. Here is how you can modify your Django LOGGING setting to do it:

# settings.py
import logging

logging.captureWarnings(True)
LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
        },
    },
    "root": {
        "handlers": ["console"],
        "level": "WARNING",
    },
    "loggers": {
        "apps": {
            "handlers": ["console"],
            "level": "INFO",
            "propagate": False,
        },
        "py.warnings": {
            "handlers": ["console"],
            "level": "WARNING",
            "propagate": False,
        },
    },
}

Integration is enabled with logging.captureWarnings(True) and use the py.warnings logger, hence the last logger definition. Here are the official docs on how to integrate that, if you're interested to read more.

The final piece to get our warning actually emitted, is to set the filter via the PYTHONWARNINGS environment variable:

PYTHONWARNINGS="all:is_user_allowed is deprecated:DeprecationWarning:"

If you need to set multiple filters in the env variable, they need be comma separated:

PYTHONWARNINGS="all:is_user_allowed is deprecated:DeprecationWarning:,all:old_function is also deprecated:DeprecationWarning:"

Conclusion

Hope this post will help you get up to speed with warnings management, the last bit to integrate with the logging module was new to me.

Keep uv.lock file up-to-date with Dependabot updates

Wed, 02 Oct 2024 00:00:00 GMT

In the past couple of weeks, there has been a lot of interest in the Python community for what started as a new Python package manager, but is now slowly growing into so much more, shipping features faster that people can blog about them.

I've been looking at adding it to some projects, but one of the main blocker is that Dependabot doesn't support it yet.

However, as the project dependencies rely on the standard PEP-621, updating these is already supported by Dependabot, and it sends some PRs for them, however, the lock file (uv.lock) is not updated automatically yet.

To workaround that, I came up with a small workflow powered by GitHub actions:

name: uv

on:
  pull_request:
    paths:
      - "pyproject.toml"

permissions:
  contents: write
  pull-requests: write

jobs:
  lock:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          token: ${{ secrets.GH_PAT }}
      - uses: astral-sh/setup-uv@v3
        with:
          enable-cache: true
      - run: uv lock
      - uses: stefanzweifel/git-auto-commit-action@v5
        with:
          commit_message: Regenerate uv.lock

Here are a few things to note:

The workflow only runs on changes to pyproject.toml, which is AFAIK the only file that could cause a lock file change.
I run the checkout with a Personal Access Token, to trigger the CI build on push. This is explained in the GitHub action I use to push.
Finally, stefanzweifel/git-auto-commit-action commit and pushes any changes back to the branch, if any.

The fact that I need to set a GitHub token in the repo secrets isn't ideal, but hopefully support will be added soon.

Alternatively, if you've not tied to Dependabot, you can use Renovate instead, which already supports the uv.lock file.

It's quite nice that we can extend GitHub features like this, the product was quite different a few years ago before actions were introduced.

Narrow state of a Django model using Python TypeGuard

Sat, 31 Aug 2024 00:00:00 GMT

Today, I came across a problem with type checking in a Django project which led me to use TypeGuard for the first time. I've heard about them in TypeScript on the Syntax podcast, so I was aware of the concept, but never had a practical situation where I needed to use them in Python.

The problem

So I was doing some refactoring to eliminate a bit of code duplication between 3 functions. As features were added, I copy-pasted the first into the second, and then later in the 3rd. I wasn't sure at the time if they were going to stay exactly the same, so copying made sense at the time. The DRY principle is well known, but optimising too early isn't good either, and I've heard some folks say that DRY should be applied when you repeat yourself 3 times, so here we are.

The code is in a Django codebase, dealing with Django models which can be in different states. As a simplified example, there was an Order model, with what the customer purchased. When the order is being prepared, a lot of fields are empty, which get filled as it's being fulfilled and shipped:

from django.db import models

class Order(models.Model):
    received_at = models.DateTimeField(null=True)
    shipped_at = models.DateTimeField(null=True)
    delivery_address = models.ForeignKey(
        Address,
        on_delete=models.PROTECT,
        related_name="orders",
        blank=True,
        null=True,
    )

My duplicated piece of code was checking for these empty fields before moving on to the main logic where the fields are needed:

def send_order_shipped_email(order: Order) -> None:
    if not (
        order.received_at
        and order.shipped_at
        and order.delivery_address
    ):
        return

    context = {
        "order_id": order.id,
        "received_date": format_date(order.received_at),
        "received_time": format_time(order.received_at),
        "shipped_date": format_date(order.shipped_at),
        "shipped_time": format_time(order.shipped_at),
        "delivery_address": order.delivery_address.short_address(),
    }

    send_template_email("emails/order_shipped.html", context)

The last line differed depending on the shipping status, but you get the idea. So I was trying to extract the pre-conditions and the building of the context. This was my initial solution:

def is_order_shipping(order: Order) -> bool:
    return bool(
        order.received_at
        and order.shipped_at
        and order.delivery_address
    )


def build_context(order: Order) -> dict[str, str]:
    if not is_order_shipping(order):
        return {}

    return {
        "order_id": order.id,
        "received_date": format_date(order.received_at),
        "received_time": format_time(order.received_at),
        "shipped_date": format_date(order.shipped_at),
        "shipped_time": format_time(order.shipped_at),
        "delivery_address": order.delivery_address.short_address(),
    }


def send_order_shipped_email(order: Order) -> None:
    context = build_context(order)
    if not context:
        return

    send_template_email("emails/order_shipped.html", context)

The tests passed, but mypy wasn't happy! In my build_context function, it complained that received_at, shipped_at and delivery_address may be None, which could happen, but I was checking for this case before using them. However, because it's done in a separate function that simply returned a bool, mypy wasn't able to infer that types were checked properly.

The solution

This is where I remembered hearing about the concept of type guards. I found the mypy docs for it, which gives a few good examples.

However, in my case, I needed to declare that the Order type had some fields checked and ensured that they were not nullable. I ended up defining a specialised order type and using it as type guard argument:

from __future__ import annotations

import datetime as dt
from typing import TYPE_CHECKING, TypeGuard

if TYPE_CHECKING:
    class ShippedOrder(Order):
        received_at: dt.datetime
        shipped_at: dt.datetime
        delivery_address: Address


def is_order_shipping(order: Order) -> TypeGuard[ShippedOrder]:
    return bool(
        order.received_at
        and order.shipped_at
        and order.delivery_address
    )

However, that didn't fully work, mypy complained that the Django fields were incompatible with the types of my specialised class:

error: Incompatible types in assignment (expression has type "Address", base class "Order" defined the type as "ForeignKey[Address | Combinable | None, Address | None]")  [assignment]
error: Incompatible types in assignment (expression has type "datetime", base class "Order" defined the type as "DateTimeField[str | datetime | date | Combinable | None, datetime | None]")  [assignment]
error: Incompatible types in assignment (expression has type "datetime", base class "Order" defined the type as "DateTimeField[str | datetime | date | Combinable | None, datetime | None]")  [assignment]

I'm not sure how to properly resolve these yet, so I marked the fields of my ShippedOrder with # type: ignore[assignment].

Conclusion

The solution isn't perfect, and it can be a bit cumbersome to define these types, I'm glad there was a solution for it. I'm glad I could take the knowledge I got from a podcast mostly focused on another language and apply it to Python.

Attest build provenance for a Python package in GitHub actions

Thu, 08 Aug 2024 00:00:00 GMT

As you may have noticed, supply chain attacks ae on the rise. These attacks usually target a small piece of software infrastructure that the target depends on and inject some malicious code. They regularly make the news and this problem is becoming evermore present in a world where software relies more and more on open source. Supply chains can be difficult to audit and reading every line of code you depend can be very time consuming at best and practically impossible at worst (good luck with your npm_modules folder). The alternative of building software without any sort of dependencies is several order of magnitude harder.

There is an initiative to try to improve the current state of affairs and aim to make the supply chain easier to audit. One of the main output as it stands is the SLSA specification, which provides guidelines to help both producers and consumers of software. I invite you to visit their website if you're interested to dive more.

I maintain a few Python packages, and while they're not used by many people, I wanted to explore what I could do as a maintainer to help the consumers of my packages. One thing that seems to gain traction is the build provenance attestation, which records where and how the package was built.

Attest build provenance action

Conveniently, there is an official GitHub action for that, which seems easy to use:

Ensure that the following permissions are set:
permissions:
  id-token: write
  attestations: write
The id-token permission gives the action the ability to mint the OIDC token necessary to request a Sigstore signing certificate. The attestations permission is necessary to persist the attestation.

Add the following to your workflow after your artifact has been built:
- uses: actions/attest-build-provenance@v1
  with:
    subject-path: "<PATH TO ARTIFACT>"
The subject-path parameter should identify the artifact for which you want to generate an attestation.

Cool, so it seems like I just need to point my wheel and source distribution at the subject path, and we're good? Let's try it!

Integration with my project

I already build my packages on CI, using PyPI's trusted publisher, and my publish job looks something like this (this is a simplified example, omitted some parts for brevity):

jobs:
  publish:
    name: Publish package
    runs-on: ubuntu-latest
    permissions:
      id-token: write
    steps:
      - name: Build package
        run: poetry build

      - name: Publish package distributions to PyPI
        uses: pypa/gh-action-pypi-publish@release/v1

The key point, is that after I run poetry build, my CI worker has a dist/ folder with the packages files, that the following step is uploading to PyPI. I use Poetry, but most Python packaging tools use this convention, I think.

Following the guide from the previous section, I get:

jobs:
  publish:
    name: Publish package
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      attestations: write
    steps:
      - name: Build package
        run: poetry build

      - uses: actions/attest-build-provenance@v1
        with:
          subject-path: "dist/*"

      - name: Publish package distributions to PyPI
        uses: pypa/gh-action-pypi-publish@release/v1

I made the change on one of my repos, and made a new PyPI release to test it, and the build finished with a new dedicated attestation section:

I also noticed that there is new page on my repo listing all the attestations.

Conclusion

I first heard about this a few weeks back and wanted to try it, and it seemed more complicated at first glance, but it was in fact pretty easy to do. I've also updated my python package template so all my projects should be using that moving forward.

I'm not entirely clear whether this will be integrated deeper into PyPI at some point, but this is still early days.

Bulk updating multiple repos with all-repos

Mon, 10 Jul 2023 00:00:00 GMT

I like the open source communities and all the developments that they enable. As a developer, I like to have the ability to solve my own problems, but when someone else already solved the problem, it's even better. However, when nobody did, I like to share my solution with others, if possible and practical. I've done it enough times that I'm beginning to have a decent setup with a project template that gives me the boilerplate I need to get started quickly, and not waste too much time packaging and publishing my solution.

The problem

However, after a while, maintaining all these projects can become overwhelming. And when external things change, like adding support for a new version of a language/framework, dropping an old one, or adapting to how a tool works, it can be tedious to do bulk updates. Of course, in this case, I'm not the first one to hit this problem, and there are a number of solutions out there.

The solution

I've been using all-repos for about a year and a half, and I'm quite happy with it. It's a tool that allows you to run a command on all or some of your GitHub repositories. It's a command line tool written in Python, that's overall quite easy to use, with an API that has a bit of a learning curve, but is very powerful. In this post, I'll show you how I use it to update my projects.

Repository setup

I like to keep all my code in git and save it in GitHub, so I've created a repo for using it. So, here we are, one more repo... to manage all my repos. One step back to move forward, I guess.

Splitting the config

The recommended usage is to create a config file, all-repos.json, that contains some config controlling behaviour, as well as your credentials.

However, often switch machine, and I wanted to store the config in source control to be able to share it more easily and only keep the credentials outside. This isn't supported by all-repos, and it seems to be an intentional design decision, and the author showed no intention to support it, so I created a package to do it: all-repos-envvar. That's now 2 more repos to maintain... I hope that's worth it!

Learning the library

The library ships several separate CLI tools, but I must be missing their point, and actually I don't even use them.

What I really find useful is the autofixer, which is documented at the bottom of the README. The documentation describes the API, and then finally gives an example autofixer, which has the boilerplate you need. The documentation then links to the built-in autofixers which are great examples on what you can do.

My usage

So in my project, I have:

A pyproject.toml file with the dependencies I need, managed by Poetry.
A all-repos.json file that contains the config for all-repos, in git. It controls how I fetch the repos, how changes are submitted (pull request), the organisations I want to include, and some repos I want to exclude.
My GitHub credentials, in a .env file that I keep outside of source control.
Finally, I copied the given boilerplate and wrote a run_fix.py file which I keep changing for my need of the day.

Example

Python 3.7 recently reached EOL, so that was a good opportunity to use the tool to update all my projects in bulk. Here is how the script looked like:

import argparse
from pathlib import Path

from all_repos import autofix_lib
from all_repos.grep import repos_matching

# Find repos that have this file...
FILE_NAME = "pyproject.toml"
# ... and which content contains this string.
FILE_CONTAINS = "tool.poetry"
# Git stuff
GIT_COMMIT_MSG = "feat: drop support for Python 3.7"
GIT_BRANCH_NAME = "feat/drop-python-3.7"


def apply_fix():
    """Apply fix to a matching repo."""
    pyproject_toml = Path("pyproject.toml")
    content = pyproject_toml.read_text()

    # Be idempotent
    if 'python = "^3.8"' in content:
        return

    new_content = content.replace(
        'python = "^3.7"',
        'python = "^3.8"',
    )
    pyproject_toml.write_text(new_content)

    ci_yml = Path(".github/workflows/ci.yml")
    if ci_yml.exists():
        content = ci_yml.read_text()
        content = content.replace(
            'python-version:\n          - "3.7"\n',
            "python-version:\n",
        )
        ci_yml.write_text(content)

    pc_yml = Path(".pre-commit-config.yaml")
    if pc_yml.exists():
        content = pc_yml.read_text()
        content = content.replace(
            '--py37-plus',
            '--py38-plus',
        )
        pc_yml.write_text(content)


# You shouldn't need to change anything below this line


def find_repos(config) -> set[str]:
    """Find matching repos using git grep."""
    repos = repos_matching(
        config,
        (FILE_CONTAINS, "--", FILE_NAME),
    )
    return repos


def main():
    """Entry point."""
    # Needed to add all-repos command line options
    # and ability to parse config
    parser = argparse.ArgumentParser()
    autofix_lib.add_fixer_args(parser)
    args = parser.parse_args(None)

    repos, cfg, commit, stg = autofix_lib.from_cli(
        args,
        find_repos=find_repos,
        msg=GIT_COMMIT_MSG,
        branch_name=GIT_BRANCH_NAME,
    )
    autofix_lib.fix(
        repos,
        apply_fix=apply_fix,
        config=cfg,
        commit=commit,
        autofix_settings=stg,
    )
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Ok that's a lot of code, so let's break it down.

Constants

Fist off, I defined a few constants at the top of the file, which I always need to change:

The FILE_... ones are used to detect whether the repository is suitable for this change: in this example, a node project would be filtered out with this mechanism, in the find_repos() function.
the GIT_... ones are for the git commit message and branch. If anything changes, the tool opens a pull request which I can review before merging.

`apply_fix()`

This is where the actual code change happens. This function runs in the context of a matching repo, and the current directory is the top of the repo. You can read and write file as you please or run commands with autofix_lib.run(...). It's a good idea to start with a check to verify that the fix hasn't run yet, just in case.

`find_repos()`

This is mainly boilerplate, it's essentially a wrapper around git grep and uses 2 of the constants I explained earlier. If you want to learn more, I suggest to go read the source code of repos_matching.

Entry point

Finally, the main() function is the entry point. It's called when running python run_fix.py, and it mostly calls the autofix API with our parameters. This is a slightly simplified version from the boilerplate.

Lifecycle of the script

I keep this script around and just change it in place, committing each time I change it to make a bulk change. It's usually some one-off operations, but it's quite useful to have some history in git in case I need to do a similar task again, like when I'll need to drop Python 3.8. You can check its history on GitHub to see how it evolved.

Trade-offs

On the downside, one might argue that the code is a bit verbose, but in my opinion it's a good thing. I'm comfortable with Python, it does the job and let me use all of its power to do the changes I need, and step into a debugger when I need to.

I came across other approaches, for example, Adam Johnson described how he uses a tool called myrepos to do something similar. It didn't fit my way of doing thing, but it might work for you.

Conclusion

I think that's it! I hope this was useful, and will help make the all-repos tool more accessible. It's quite powerful, although I found that it had lots of things I didn't need.

Nested ViewSets with Django REST Framework

Fri, 05 May 2023 00:00:00 GMT

I've been using Django REST Framework for a while now but mainly in old projects where lots of code have been written. In most cases, these projects didn't make good use of ViewSets, due to a variety of reasons. It may be due to the original author not knowing about them (that was me at some point), or because the operation behind each HTTP verb needs to be specialised or simply that a single operation is needed. However, in a recent project, I got the opportunity to start a lot of new code and make the most of them. So despite having used DRF for years, I feel like I'm learning a lot of new tricks.

import { Callout } from "../../../components/callout";

<Callout level="note"> This post used to be a much shorter version as "TIL", but a few days after writing it, I found a much more elegant solution, which turned into a longer form post. The first solution was the TIL. </Callout>

The problem

In a project, I needed to add multiple HTTP verbs to a nested resource in a ViewSet. Let's take an example to illustrate. Let's imagine an application to manage some galleries of photos. The galleries need to be only accessible to the user that ows them. Assuming the models look like this (simplified for brevity):

class Gallery(models.Model):
    user = models.ForeignKey(User, on_delete=models.CASCADE)


class Photo(models.Model):
    gallery = models.ForeignKey(Gallery, on_delete=models.CASCADE)

I needed the following API endpoints:

GET    /galleries/
POST   /galleries/
GET    /galleries/:id/
GET    /galleries/:id/photos/
POST   /galleries/:id/photos/
GET    /galleries/:id/photos/:photoId/
PUT    /galleries/:id/photos/:photoId/
PATCH  /galleries/:id/photos/:photoId/
DELETE /galleries/:id/photos/:photoId/

I already had a ViewSet for the getting and creating galleries, meaning the first 3 endpoints existed. I now needed to add photos management to it. I tried to look for how to nest a ViewSet inside another, but I couldn't find anything supported out of the box.

The first solution

I knew about the @action decorator, and while looking closer at the DRF docs, I found out about how to route multiple HTTP verbs to an existing action, which I didn't before. This seemed pretty clean, so I decided to give it a go. I ended up with the following:

class GalleryViewSet(viewsets.ModelViewSet):
    queryset = Gallery.objects.all()
    serializer_class = GallerySerializer
    permission_classes = (IsAuthenticated,)

    def get_queryset(self):
        return self.queryset.filter(user=self.request.user)

    @action(
        detail=True,
        methods=['get'],
        url_path='photos',
        url_name='photos-list',
        serializer_class=PhotoSerializer
    )
    def list_photos(self, request):
        gallery = self.get_object()
        photos_qs = gallery.photos.all()
        serializer = self.get_serializer(photos_qs, many=True)
        return Response(serializer.data)

    @list_photos.mapping.post
    def create_photo(self, request):
        ...

    @action(
        detail=True,
        methods=['get'],
        url_path='photos/(?P<photo_id>[^/.]+)',
        url_name='photos-detail',
        serializer_class=PhotoSerializer
    )
    def get_photo(self, request, pk=None):
        ...

    @get_photo.mapping.patch
    def update_photo(self, request, pk=None):
        ...

    @get_photo.mapping.delete
    def delete_photo(self, request, pk=None):
        ...

I liked how it was declarative and how each HTTP verb was separated into its own method. However, it's quite a lot of code, with some bits duplicated, especially each method body. I also needed to check ownership of the gallery. It felt like I was re-implementing all the code for a photo ViewSet as individual methods.

Also, I didn't realise at the time, but all these actions are using the QuerySet from the base ViewSet. This is a simplified example, but in my case, the base ViewSet had several select_related and prefetch_related to optimise the fetching of the gallery, but these were not needed for the photo management part. So I was fetching a lot of data that I didn't need.

The second solution

After a couple of days, while not being happy with my solution, I decided to give it another go. I noticed while implementing my initial solution that I could add extra URL parameters in the url_path of the get_photo method, which made me think that maybe I could do the same when registering the ViewSet, and sure enough, I could! So I ended up with the following:

# views.py
class GalleryViewSet(viewsets.ModelViewSet):
    queryset = Gallery.objects.all()
    serializer_class = GallerySerializer
    permission_classes = (IsAuthenticated,)
    lookup_url_kwarg = "post_id"

    def get_queryset(self):
        return self.queryset.filter(user=self.request.user)

    # Note: Removed all the actions methods for photos


class PhotoViewSet(viewsets.ModelViewSet):
    queryset = Photo.objects.all()
    serializer_class = PhotoSerializer
    permission_classes = (IsAuthenticated,)
    lookup_url_kwarg = "photo_id"

    def get_queryset(self):
        return self.queryset.filter(gallery_id=self.kwargs["gallery_id"])

    def perform_create(self, serializer):
        serializer.save(gallery_id=self.kwargs["gallery_id"])



# routers.py
router = SimpleRouter()
router.register(
    "galleries",
    GalleryViewSet,
    basename="gallery",
)
router.register(
    "galleries/(?P<gallery_id>[^/.]+)/photos",
    PhotoViewSet,
    basename="gallery-photo",
)

As a bonus, I added a lookup_url_kwarg to both ViewSets, to make the URL parameters more consistent and self-explanatory in the API docs. This solution is much more DRY and use the full power of DRF. I was pretty satisfied with it.

Bug: access control

However, I soon realised that this solution had a bug. The get_queryset method of the PhotoViewSet was no longer checking the ownership of the gallery! This meant that a user could access photos from galleries that they don't own, and even create photos in them! This was a big no-no. I tried to write a simple method to fetch the gallery and check ownership, but I wanted to do it pretty early and for all methods, including create(). I thought of overriding the dispatch method, but it felt like there wasn't a good hook for that.

After a bit of thinking, I realised that I needed to control the user had permissions to access the gallery, and DRF has a built-in solution for that, with permissions classes. I implemented my own permission class, which looked like this:

class UserOwnsGallery(permissions.BasePermission):

    def has_permission(self, request, view):
        gallery_id = view.kwargs.get("gallery_id")
        if gallery_id is None:
            return False

        get_object_or_404(
            Gallery,
            id=gallery_id,
            user_id=request.user.id,
        )
        return True

If the user doesn't own the gallery, I'm returning a HTTP 404 as I think it's a security best practice. If I were to return "403 Forbidden" instead, I would reveal some information to a potential attacker, that the gallery with the ID exists. It's probably not a big risk in our application, but it's best to be safe.

And here is how I use it on my ViewSet:

class PhotoViewSet(viewsets.ModelViewSet):
    queryset = Photo.objects.all()
    serializer_class = PhotoSerializer
    permission_classes = (
        IsAuthenticated,
        UserOwnsGallery,
    )
    lookup_url_kwarg = "photo_id"

Conclusion

I'm pretty happy with the final solution. It's DRY, it's using the full power of DRF, and it's secure. I'm not sure if it's the best solution, but it's probably very close from it. I hope this article will help you if you're in a similar situation.

Migrate my site design system to Chakra UI with ChatGPT

Thu, 27 Apr 2023 00:00:00 GMT

A short story on how I migrated this site from a custom design system to Chakra UI with the help of ChatGPT. Wasn't a complicated migration, but AI was a great help to get it done quicker and discover their great documentation.

Initial state

Before I started, this site was using a custom design system based on styled-components. It was a small library of components that I had created for a few projects almost 3 years ago, in 2020. Back then, the world was in lockdown, and I had lots of time at home to toy around some fun little projects. I also wanted to work on my CSS skills and didn't need many components, but I needed them on a few projects. Instead of copying them around, I decided to created my own mini design system with a small React component library. This site was one of the first to use it.

Problems

Creating a design system requires to have some basic knowledge in design theory and typography to get spacing right while giving flexibility to the consumers of the library. I didn't have that knowledge back then, and I still don't have it now. I did some research and learnt by taking some inspirations in other projects to learn, but it was and remain a very basic design system. It looked ok, but I'm sure someone more proficient in design would spot a lot of problems.

I migrated the 3 or 4 sites I had to use that library and didn't touch it since then. Recently (~2 years later), I wanted to add a new section to this site, with a different layout. Shortly after I started, I realised that I didn't remember the syntax my own self created a few years back, and it was barely documented in my design system. Sure, I could work it out by looking at the code, but it didn't work when I tried: this site wasn't configured exactly how the library was expecting. Basically, it was as if I was using someone else's broken library, and it wasn't fun anymore!

Solution

Writing my own design system was a fun exercise, but it was time to move on. I wanted to use a mature library, with good documentation, widely used in the React community and which worked with my toolchain (Gatsby.js). I heard a few times of Chakra UI, and it seemed to tick all the boxes, and a lot more. I used to see "doing too much" as a drawback, but today it's no longer the case, I just want something that works.

On top of the requirements I needed, I also got sold on these features specifically:

Themeable design system
CSS in JS
Provides Atomic components as well as complex ones

Challenges

However, migrating came with its own set of challenges. The New syntax, different from styled-components, would take some time to get used to. The library, with all its component was a bit overwhelming at first, it looked like there was a component for everything, and like many components could be used to do the same thing.

How AI can help

That challenge seemed like a perfect job for ChatGPT: lots of data to crunch though, lots of content online to back it up. Let's go!

Migration

Component level

I decided to start with some self-contained components that I wrote myself. Just asked to convert from styled-components to Chakra UI with the code below and it spitted out pretty quickly a first version.

Input:

import styled from "styled-components";

export const PostMetaData = styled.div`
  display: flex;
  flex-direction: column;
  justify-content: space-between;
  align-items: center;
  margin-bottom: 1rem;

  @media (min-width: 400px) {
    flex-direction: row;
  }

  div {
    margin-bottom: 0.5rem;
  }
`;

Output:

import { Box, useBreakpointValue } from "@chakra-ui/react";

export const PostMetaData = (props) => {
  const flexDirection = useBreakpointValue({ base: "column", md: "row" });

  return (
    <Box
      display="flex"
      flexDirection={flexDirection}
      justifyContent="space-between"
      alignItems="center"
      marginBottom="1rem"
      {...props}
    >
      {React.Children.map(props.children, (child) => (
        <Box marginBottom="0.5rem">{child}</Box>
      ))}
    </Box>
  );
};

It didn't produce totally equivalent code, it actually looked better with the breakpoints it gave me. I didn't really need to deal with children, but by handling this case that, it showed me some flaws in my original design. Off to a great start! I tweaked a bit its output and moved on to more transformations. Some were pretty trivial and barely needed any changes.

When a new Chakra component showed up, I looked in their documentation to learn about it and how it could be used. I also looked at the code it produced to see how it was used. Repeating this, I quickly discovered the main components I needed and after a few migrations I felt like I could write them myself, but it was still faster to ask the AI.

Limitation

One migration didn't work too well, it was the Avatar component of this site. Here is the original version:

const AvatarStyles = styled.div`
  padding: 1rem;

  img {
    padding: 5px;
    border: 1px solid ${(props) => props.theme.grey};
    border-radius: 50%;
  }
`

export const Avatar: React.FC = () => (
  <AvatarStyles>
    <StaticImage
      src="../assets/avatar.jpg"
      alt="Picture of Bruno"
      placeholder="blurred"
      layout="fixed"
      width={160}
      height={160}
    />
  </AvatarStyles>
)

The AvatarStyles was using a nested rule for an img tag which is inside the StaticImage component. The CSS rule was targeting a non-direct descendant, which ChatGTP initially wrote using an underscore-prefixed prop _img:

export const Avatar: React.FC = () => (
  <Box
    padding="1rem"
    _img={{
      padding: '5px',
      border: '1px solid',
      borderColor: 'gray.500', // Adjust the color according to your theme
      borderRadius: '50%',
    }}
  >
    <StaticImage
      src="../assets/avatar.jpg"
      alt="Picture of Bruno"
      placeholder="blurred"
      layout="fixed"
      width={160}
      height={160}
    />
  </Box>
);

After telling it that it doesn't work, it tried something else, but just failed to produce an output that worked. It seems that this might be an old syntax and the AI didn't know the newer syntax from the version I used. I eventually landed on this:

export const Avatar: React.FC = () => (
  <Box
    padding={4}
    sx={{
      img: {
        borderRadius: '50%',
        padding: 'var(--chakra-space-1)',
        border: '1px solid',
        borderColor: 'gray.500',
      },
    }}
  >
    <StaticImage
      src="../assets/avatar.jpg"
      alt="Picture of Bruno"
      placeholder="blurred"
      layout="fixed"
      width={160}
      height={160}
    />
  </Box>
)

Theme / global styles

The theme is done via a Javascript object, passed to the ChakraProvider component wrapping the whole app. The global styles are included automatically from there. Pretty straightforward to use. The thing I really liked is that everything is in one place, in my theme object, while before, I had things split between my theme and global styles, and they were 2 separate files.

What took me a bit of time to figure out what keys/values are available. Once I understood that keys are mapped to CSS variables, it clicked and I could find what to customise by looking at the CSS variables in the browser.

Results

The resulting pull request is huge, but ChatGPT helped my and it was actually pretty painless. I think the design looks very close, but is overall improved. If/when I need something new, I have plenty of battle-tested components to choose from. I also ran Lighthouse to make sure there wasn't any performance regression, and it actually improved a bit.

So overall, really happy with that migration, and I'm looking forward to using Chakra UI in my next projects.

Enabling sudo via TouchID using Ansible

Fri, 25 Nov 2022 00:00:00 GMT

I discovered fairly recently in a blog post that it was possible to enable sudo mode in the terminal via Touch ID instead of having to type my password. I'm glad I wrote a TIL about it because I was able to find it again very easily and do it again on a new machine.

Today, I went a step further and integrated this into my Ansible playbook that I use to provision my machine. This was a pretty simple action which made me learn a bit more about Ansible. If you're in a rush, here is the commit with the solution I ended up with:

# Do sudo using Touch ID
- name: Set content of sudo file as fact
  set_fact:
    sudo_conf: "{{ lookup('file', '/etc/pam.d/sudo') }}"
  ignore_errors: yes
  tags:
    - touchid

- name: Set sudo via Touch ID if not setup
  become: yes
  command: sed -i '' 's/auth       sufficient     pam_smartcard.so/auth       sufficient     pam_smartcard.so\nauth       sufficient     pam_tid.so/g' /etc/pam.d/sudo
  when: "'auth       sufficient     pam_tid.so' not in sudo_conf"
  tags:
    - touchid

Here is the breakdown of what's going on, with the key learning:

I use the set_fact module to read the content of the file and store it in a sudo_conf variable.
The following task uses the when condition to check if the sudo_conf variable (and therefore the file) already contains the required string. If it does, it will not run the task.
If the string isn't present, I use sed to add it in the right place.
Use sed's -i option to edit the file in place. The version of sed shipping with macOS has a quirk and needs an empty string to edit in place (means no suffix). An alternative is to brew install gnu-sed and use gsed -i without the following empty string.
Use become: yes to run the command as root.

And that's it, short and sweet. I'm not sure if this is the best way to do it, but it works and I learned a bit more about Ansible.

Specify docs dependency groups with Poetry and Read the Docs

Mon, 21 Nov 2022 00:00:00 GMT

I wrote a while back on how to migrate a project to Poetry. One of the gotcha was how to specify the dependencies to build the project documentation, especially on Read the Docs. The solution, at the time, was to specify these dependencies as "extras", which had the unintended side effects of being visible to your users, while only needed for internal purposes. By that, I mean that the package could be installed with pip install package-name[docs].

Dependency groups

Poetry 1.2 introduced a new feature which I have been waiting for a while: dependency groups. Until recently, the only groups were the main dependencies (the one that your package needs to run) and the development dependencies (the ones that are only needed for development, like testing or linting).

With dependency groups, you can now specify any arbitrary group of dependencies, and use them in any way you want. they can be optional or not.

This help us solve my previous problem, now I can add an optional docs dependency group and use it to build the documentation. I use MyST parser with Sphinx, so here is how it might look like:

[tool.poetry.group.docs]
optional = true

[tool.poetry.group.docs.dependencies]
myst-parser = ">=0.16"
sphinx = ">=4.0"
sphinx-autobuild = ">=2021.0"
sphinx-rtd-theme = ">=1.0"

Now I can install my docs dependencies with:

poetry install --with docs

And no extras are added to my package. Nice.

Read the Docs

The other piece of the puzzle is to get my documentation online. I usually do this using Read the Docs. They have good support for installing the package being documented with extras, but they don't have Poetry installed, so it needs a bit of customisation. The good news is that their config file is flexible enough to allow it. The example for Poetry was just a bit out-of-date with the recent Poetry 1.2 release. I sent a pull request to fix it, but came up with a suboptimal solution, which one of the maintainer helped me improve. The updated version will be published soon, but here is what I ended up with:

version: 2

build:
  os: ubuntu-20.04
  tools:
    python: "3.9"
  jobs:
    post_create_environment:
      # Install poetry
      - pip install poetry
      # Tell poetry to not use a virtual environment
      - poetry config virtualenvs.create false
    post_install:
      # Install dependencies
      - poetry install --with docs

sphinx:
  configuration: docs/source/conf.py

The main trick is to configure Poetry to not create virtual environment, and then pass the poetry install command form the previous section with docs group as post_install. RTD doesn't let you customise the installation step, and it may install some older version of Sphinx, so by setting as post_install, you're sure to get what you ask for. This last bit shouldn't be a problem unless your RTD project was created a long time ago.

Closing words

I hope this helps you if you're using Poetry and Read the Docs. If you have any questions, feel free to reach out to me on Mastodon or Twitter.

Convert a Poetry package to the src layout

Wed, 19 Jan 2022 00:00:00 GMT

The src layout is commonly used in Python ecosystem nowadays (even the pytest docs recommend it), but I'm not really using it on my projects. I think I've never hit the pain points it solves, and last time I tried, it made my development workflow more complicated.

I'm not going into the whys, they are detailed in this excellent blog explaining its rationale in detail, where I read about it the first time.

In recent years, as it became more popular, the tooling has caught up, and I wanted to try it again to see if the situation improved.

Starting a new project

If you start a new project, Poetry supports it out of the box in their poetry new command:

poetry new --src my-package

Although you might want to use a full blown template with other tools also configured (e.g. pytest).

Converting a project

I found it a bit less well documented how to convert a project, so here you go. Assuming the project uses Poetry, pytest and a layout as follows:

my-package
├── poetry.lock
├── pyproject.toml
├── my_package
│   ├── __init__.py
│   └── main.py
└── tests
    ├── __init__.py
    └── test_main.py

Here are the steps I followed to convert to src/ layout, without changing the imports in my tests:

Create a src/ folder. And I really mean folder, not package: do not create a __init__.py in that folder.
Move my_package into src/.

Change you packages section in your pyproject.toml:

  packages = [
-     { include = "my_package" },
+     { include = "my_package", from = "src" },
  ]

Add pythonpath to your pytest option as follows:
```
  [tool.pytest.ini_options]
+ pythonpath = ["src"]
```
If using pytest version lower than 7, you'll also need to install pytest-srcpaths:
```
poetry add -D pytest-srcpaths
```

If your pyproject.toml references some paths, make sure to update them, e.g python-semantic-release:

  [tool.semantic_release]
  branch = "main"
- version_variable = "my_package/__init__.py:__version__"
+ version_variable = "src/my_package/__init__.py:__version__"

If everything went well, you should have the following layout and running pytest should work:

my-package
├── poetry.lock
├── pyproject.toml
├── src
│   └── my_package
│       ├── __init__.py
│       └── main.py
└── tests
    ├── __init__.py
    └── test_main.py

Conclusion

This was much easier than I remembered! I think I'll change all my projects to this layout, having this guide will help me reprodude it. Hopefulluy it'll help you as well!

Convert Python requirements to Poetry format

Thu, 07 Jan 2021 00:00:00 GMT

I recently wrote about migrating a Python project to Poetry, however, this guide is quite manual process. I recently came across dephell, which seems to be able to automate some part of it. I wanted to check out how it might help for this follow-up post.

Migrate setup.py stuff

First step is to migrate meta-data and direct dependencies specified in setup.py file:

dephell deps convert \
  --from=setup.py \
  --to-format=poetry \
  --to-path=pyproject.toml

This converts some package metadata and add the dependencies which are listed in your setup.py.

Convert development dependencies

The development dependencies are located in requirements.in (as I used pip-tools), and to not overwrite the dependencies from the previous step, let's write them to a temporary file:

dephell deps convert \
  --from=requirements.in \
  --to-format=poetry \
  --to-path=pyproject-dev.toml

Create a new section for development dependencies in pyproject.toml and move over dependencies from pyproject-dev.toml:

[tool.poetry.dev-dependencies]
# copy deps from pyproject-dev.toml here

Now you can get rid of the temporary pyproject-dev.toml file.

Finish

As far as I'm aware, that is as far as this tool will get you. The pyproject.toml file should almost be ready for Poetry, you can try running poetry install and fix the remaining issues.

In my case, here are the things I needed to adjust:

the minimum Python version wasn't properly detected, it was just '*' despite my setup.cfg having python_requires >= 3.6.
Not all metadata were converted, I had to finish them manually.
Some extra classifiers were included although Poetry can generate some of them automatically.
Other tools configured via setup.cfg which I wanted to migrate to pyproject.toml are not migrated by this tool, it's -understandably- out of its scope.

Nevertheless, this tool could speed up migration for projects with lots of dependencies.

Migrating a project to Poetry

Sun, 18 Oct 2020 00:00:00 GMT

Poetry is a tool solving the problem of Python packaging. It was started back in February 2018 by Sébastien Eustace (also the author of pendulum). It has a beautiful website and an ambitious headline:

Python packaging and dependency management made easy

It has been on my radar for a while, but I never gave it a proper go. I was happily using pip-tools, which was solving my main use case, while being a lot more lightweight and meant I could keep working with pip since its output is a good old requirements.txt. I heard about it a few times online, often next to Pipenv, but recently it looks like Poetry got a bit more traction.

Having a bit of time on my hands, a few weeks ago I decided to take a proper look at it and maybe migrate one of my projects, Deezer Python.

Starting point

Before the migration, here is how the package was managed:

Project metadata were in setup.cfg, using setuptools declarative config.
Development (and documentation) dependencies managed by pip-tools.
Documentation hosted on Read the Docs (RTD).
Releases automated with Python Semantic Release (PSR) on Github Actions.
Development tools configured in setup.cfg (black, isort, pyupgrade, flake8).
Using tox to help local testing.
Using Github Actions for CI.

These are a couple of features that were impacted by migrating to Poetry, and I think it's worth mentioning them for context. Since Poetry uses pyproject.toml, I was also hoping to move all my tools config from setup.cfg to that file.

Migration

Update: I recently came across the dephell tool which seems to automate some conversions check out the follow-up post.

Installing Poetry

The first step is to get the CLI. I initially installed it via Homebrew, but later realised that Poetry was setting some default values based on the Python version its installation uses. As the Homebrew Python can be updated without notice, I realised it was not the best option here, so I later reinstalled it via pipx using a Python installation managed by pyenv that wouldn't be wiped without my knowledge:

pipx install \
 --python ~/.pyenv/versions/3.8.6/bin/python \
 poetry

Project metadata

The first step was to migrate the project metadata from setup.cfg to pyproject.toml. Poetry comes with a handy interactive command poetry init which will create a minimal pyproject.toml for you. I already noticed a few pleasant surprises:

The CLI was very nice to interact with
The author and author_email from setup.cfg were merged into an array of authors, each with the format Full Name <email@address.com>.

I then went on to convert more settings into the new format manually, and the process was quite painless. Many settings have the same name and values, and when they are different, it's mainly to simplify things. I guess it's something a library like setuptools cannot easily afford to do due to backwards compatibility, but that a new opt-in tool like Poetry can.

Dependencies

Adding dependencies and development dependencies was pretty simple, I just needed to run poetry add [-D] ... with the list of packages at the end.

In this process, I discovered that one of the development dependencies, pyupgrade is not compatible with Python 3.6.0: Poetry would not let me set my own Python to ^3.6. I initially changed my minimum version to ^3.6.1 as a quick fix, but I realised later that it impacted my package's trove classifiers, the ones generated by Poetry: my package wasn't listed as Python 3.6 compatible. Since this is just for a development dependency, there is a better way! The fix is to specify the dependency conditional to the Python version:

[tool.poetry.dependencies]
python = "^3.6"
...

[tool.poetry.dev-dependencies]
...
pyupgrade = { version = "^2.7.3", python = "^3.6.1" }

With this, the minimum Python version of my package and its classifiers are correct.

Extra Dependencies

This package had an "extra require" dependency. There is a good example in the documentation, these dependencies need to be specified in the same section as normal dependencies, but as optional:

[tool.poetry.dependencies]
...
tornado = {version = "^6.0.4", optional = true}

And a dedicated pyproject.toml section maps the extras to an array of optional dependencies:

[tool.poetry.extras]
tornado = ["tornado"]

I got confused initially because I thought it was done via the poetry add ... -E ... command. However, it does something different, it's for the extras of the dependency, not the extras of my own library the dependency should fall under.

For example Django has 2 possible extras at the moment, so one would run the following command to use them:

poetry add Django -E argon2

It means "install Django with the argon2 extra". I thought it meant "install Django and put it under the argon2 extra".

Docs dependencies

The dependencies to build the docs were specified in a requirements.txt in the docs/ folder and RTD was configured to pick this up. I initially thought that I wouldn't be able to remove that file, but it turns out it's possible to make it work.

Thanks to PEP 517, which Poetry is compliant with, you can do pip install . in a Poetry package. This has been in pip since 19.0, and pip running on RTD is newer than this. However, this method wouldn't install your development dependencies, so your docs dependencies cannot be specified as such. It works if you specify a docs extra, though:

# pyproject.toml
[tool.poetry.dependencies]
...
myst-parser = {version = "^0.12", optional = true}
sphinx = {version = "^3", optional = true}
sphinx-autobuild = {version = "^2020.9.1", optional = true}
sphinx-rtd-theme = {version = "^0.5", optional = true}

[tool.poetry.extras]
...
docs = [
    "myst-parser",
    "sphinx",
    "sphinx-autobuild",
    "sphinx-rtd-theme",
]

# readthedocs.yml
version: 2
python:
  install:
    - method: pip
      path: .
      extra_requirements:
        - docs

The downside of this, is that the extra is part of your package, so not ideal.

Releases

I recently moved the automation of releases to Python Semantic Release which worked well for me, and this would have been a blocker if it hadn't worked. These are the pieces I needed:

Move its config to come from pyproject.toml.
Package version is specified in pyproject.toml as well as in __init__.py.
Update build command to use Poetry instead of setuptools.

Here is the PSR config in pyproject.toml to achieve that:

[tool.semantic_release]
version_variable = [
    "deezer/__init__.py:__version__",
    "pyproject.toml:version"
]
build_command = "pip install poetry && poetry build"

I was expecting to have to change more, but it all worked out of the box with just that.

Linting and code formatting

All the tools I use for linting and code formatting were configured via setup.cfg and ideally I'd like to replace it by pyproject.toml. It was possible for almost everything, except for flake8 which has an open issue for it.

I decided to move as many things as I could to pyproject.toml, and move flake8 config to .flake8.

With all the above, I was able to remove the setup.cfg as well as all the pip-tools files for dependencies.

Code Coverage

Pytest had an unexpected config section for pyproject.toml, I didn't pay much attention to it initially and put its config under [tool.pytest], while the section should actually be [tool.pytest.ini_options].

The consequence of this was that my config was ignored and I didn't run the tests with coverage enabled, which silenced another error in the coverage section tool.coverage.run, where source should be an array:

[tool.pytest.ini_options]
addopts = "-v -Wdefault --cov=deezer"

[tool.coverage.run]
branch = true
source = ["deezer"]

With this, pytest was running with coverage, but for some reason, codecov wasn't picking up the report properly. I noticed that it used to find an xml file when it was working, so I edited the command on CI to generate it with --cov-report=xml:

poetry run pytest --cov-report=xml

With these changes I eventually got my code coverage reporting back, but it's something I missed in the original migration. Not directly related to Poetry, but interesting that Pytest decided to go with this section name.

Tox

Poetry works nicely with Tox, I followed the section in their FAQ, and here is an overview of the changes:

[tox]
isolated_build = true
envlist = py36,py37,py38,py39,pypy3,docs,lint,bandit

[testenv]
whitelist_externals = poetry
commands =
    poetry install
    poetry run pytest
...

I replaced the deps section in each testenv by a poetry install into the list of commands to run, and prefixed all commands to be run in the isolated environment by poetry run.

Github Actions

Poetry isn't installed out of the box on Github Actions, one could either install it with a simple run step or use a dedicated action for it. I've opted for the dedicated action, thinking that Dependabot could keep it up to date for me.

The rest of the changes are pretty simple, it's a matter of replacing pip install by poetry install -E ... and prefixing all commands by poetry run. My docs were tested and I was changing directory with cd, I took this opportunity to instead use working-directory key to the Github action step.

Verdict

Did Poetry deliver on its ambitious tagline? I think so, I was really impressed by the developer experience of Poetry, its CLI is really nice, and I hit few issues on the way. Overall the migration was not too difficult, you can check the pull request on Github. I feel like there are quite a few features where I only scratched the surface (like multi-environments). I'm going to wait a bit to see how this works in the longer run, but I think I'll migrate my other projects soon.

Auto-update pre-commit hooks with GitHub Actions

Sun, 12 Jul 2020 00:00:00 GMT

Pre-commit hooks are great to reduce the feedback loop for things like linting and auto-formatting. Git supports them out of the box, but they are not easy to share across all developers working on a project, they need to be installed by each developers.

Several tools exist to solve this problem, but my favorite is pre-commit.com. It's written in Python, but it aims at being language agnostic. It saves your setup in a config file and developers can install all of them with a single command.

Each tool is referenced by their github repo and tag to install, which is great because each tool is pinned to a specific version. However, I usually have the tool versions already elsewhere in my repository, for example in requirements.txt, causing some duplication. The main project dependencies are automatically updated with Dependabot, PyUP or Renovate but none of these tools support the pre-commit config file. After a while, it's easy to end up with versions discrepancies.

UPDATE: since this article was written, Renovate added support for updating your pre-commit hook config.

That is until this week-end, where I stumbled upon the autoupdate command from pre-commit. I'm not sure how I missed this before, it looks like it's been part of pre-commit for a really long time. By combining this with the power of Github actions, I was able to get it to send me a pull request each time a new version is available:

name: Pre-commit auto-update

on:
  schedule:
    - cron: "0 0 * * *"

jobs:
  auto-update:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Set up Python
        uses: actions/setup-python@v2
        with:
          python-version: 3.8

      - name: Install pre-commit
        run: pip install pre-commit

      - name: Run pre-commit autoupdate
        run: pre-commit autoupdate

      - name: Create Pull Request
        uses: peter-evans/create-pull-request@v2
        with:
          token: ${{ secrets.CPR_GITHUB_TOKEN }}
          branch: update/pre-commit-autoupdate
          title: Auto-update pre-commit hooks
          commit-message: Auto-update pre-commit hooks
          body: |
            Update versions of tools in pre-commit 
            configs to latest version
          labels: dependencies

This workflow is scheduled every day at midnight, runs pre-commit autoupdate and sends a pull request if there are any changes.

The piece that required a bit of fiddling is the action creating the pull request, partly to get commit message, title, content and labels right, but mostly because I initially used secrets.GITHUB_TOKEN as token, but it wouldn't trigger the CI build for that pull request.

It's a limitation which is well documented on the action's README, and is intentional from Github. I chose the solution to create a PAT scoped to repo and added it to the secrets as CPR_GITHUB_TOKEN. It's deployed and running on the repo of django-codemod.

The pull request action has fixed inputs, so it will create one pull request at a time for all updates. If several tools get a new version, they would all be updated at once, and if a pull request already exists, it would receive more updates. This is not necessarily a bad thing, but if one tool breaks the build due to new linting rules, all are stuck.

Maybe I'll look into making the pull request content a bit more dynamic, but for now it does the job I need to. I'm also planning to add this to my Cookiecutter template for Python package, so I can get it for all my new projects.

I hope this can help folks keep their pre-commit hook up to date, maybe this will become obsolete when pre-commit CI is ready, or maybe it will be a cheaper and simpler alternative

Mastering flexbox with Flexbox Zombies

Sun, 03 May 2020 00:00:00 GMT

Flexbox is not exactly new, and I've played with it a few times, but as a developers who spends most of my time in the back-end building API, I don't really need to know it at work. I do a bit of CSS here and there, but not enough to internalise it. I never took the time to really understand what various properties do, when to apply them on container vs when to apply them on content and I always had to look up the cheatsheet to do anything with it.

This was until a few weeks ago when I read about Flexbox Zombies somewhere (probably on Twitter). I'm a big fan of gamification, it hooks me up and I learn very well with this. Decided to take advantage of these unusual times where we have to stay home more than normal, I took the class and set myself to do one lesson a day.

The course is made of 12 stages -11 lessons and 1 final test- each taking between 15 to 30 mins. The first part of each lesson is explaining one or two properties and its values, followed by series of exercises with more of less help. The final test is where you need to defeat the boss and is without hints.

The result for me is that 12 days later, I feel very much like I know what I am doing when working with flexbox. Before it was like a guessing game, very much empirical, applying properties half-randomly on the container or items. Now I know what I should use for my use case. Right now, all the things I learnt are fresh but I'm confident this time it will stick in my head.

If this feel like something you'd like to learn, I would encourage you to sign-up for the course, it's free (at least now).

Thanks to Dave Geddes for putting this amazing learning content together!

Self-host your Typography.js fonts with Gasby

Sun, 05 Jan 2020 00:00:00 GMT

Problem

I recently added a CSP to this blog, and one of the tricky parts when implementing this was that the fonts were coming from from Google fonts. Several extra directives were required, and depending on the browser I was using, it was not even working properly, I think Chrome wasn't properly loading them, while Firefox was.

Investigation

To fix this, I decided that I should self-host the fonts I need instead of linking to several Google domains. I was ready to download the file and copy them in my codebase, but then I realised there is no CSS/SASS, so nothing linking to them...

I'm using the Funston theme for Typography.js with the appropriate Gatsby plugin. The fonts are specified by the theme and the appropriate link tag are injected in the HTML when needed.

The only mention of self-hosting is in the Gatsby plugin options:

omitGoogleFont: (boolean, default: false) Typography includes a helper that makes a request to Google's font CDN for the fonts you need. You might, however, want to inject the fonts into JS or use a CDN of your choosing. Setting this value to true will make gatsby-plugin-typography skip the inclusion of this helper. You will have to include the appropriate fonts yourself.

However, how to include them yourself is left unanswered, but luckily someone answered this question on Stackoverflow.

Solution

Ok, looks like I got all the pieces of the puzzle, let's code!

First, let's disable Google font from the gatsby-plugin-typography config:

// gatsby-config.js
{
  ...
  plugins: [
    {
      resolve: `gatsby-plugin-typography`,
      options: {
        pathToConfigModule: `src/utils/typography`,
        omitGoogleFont: true,
      },
    },
  ]
}

The 2 fonts I need are "Patua One" and "Cabin Condensed", and -following the previous Stackoverflow post- you can find on npm here and here:

yarn add typeface-patua-one typeface-cabin-condensed

Then add them to the onInitialClientRender browser API:

// gatsby-browser.js
exports.onInitialClientRender = () => {
  require("typeface-patua-one");
  require("typeface-cabin-condensed");
};

And that's pretty much it, now the fonts are self-hosted.

Cleaning up my CSP

As I said in the intro, the main driver for this change was my CSP, which was at best overly complicated, and at worst wrong. So now that it doesn't need to be so complicated, let's review it.

The usage of Google fonts require -at least- the following to work:

'style-src': fonts.googleapis.com fonts.gstatic.com
'font-src': fonts.gstatic.com
'connect-src': fonts.googleapis.com
'prefetch-src': fonts.googleapis.com

Some directives, like font-src and prefetch-src are only required for Google fonts, so I can remove them from the plugin config.

Final words

If you want to look at the entire change, I made a pull request for it, feel free to have a look to it.

Setting up a Content Security Policy with Gatsby

Thu, 14 Nov 2019 00:00:00 GMT

This is a quick story telling how I set up a content security policy on my personal blog which is powered by Gatsby. I've been keen to try adding CSP for a while but I know it can be tricky to get right and cover everything. However, this blog is a very simple use case as I embed very little third party scripts but it took me some time to get it right nevertheless.

First attempt

Gatsby is relatively young but has already a rich plugin ecosystem. So the first thing I did was to research any plugin that would help with this. I quickly found gatsby-plugin-csp doing exactly what I wanted. I went ahead and installed it without any options to give it a try:

// gatsby-config.js
module.exports = {
  plugins: [`gatsby-plugin-csp`],
};

I opened a pull request to check a production-like deploy preview on Netlify, but it looked pretty broken:

Looks like a few things were missing... Let's customise this plugin!

Starting to customise

The plugin has a few options, I quickly glanced the documentation and immediately started adding directives for the errors that were reported. I'm using Google analytics and Google fonts, I need to whitelist them:

// gatsby-config.js
module.exports = {
  plugins: [
    {
      resolve: `gatsby-plugin-csp`,
      options: {
        directives: {
          "script-src": `'self' www.google-analytics.com`,
          "style-src": `'self' 'unsafe-inline' fonts.googleapis.com`,
          "img-src": `'self' data: www.google-analytics.com`,
        },
      },
    },
  ],
};

Pushed to my branch again, waited for the deployment to be updated, but it was still not looking great:

Hum, weird it looks even worse! Ok the errors are different, let's keep customising. Looking at the errors, I can see another Google font domain and inlines are missing, let's add them:

// gatsby-config.js
module.exports = {
  plugins: [
    {
      resolve: `gatsby-plugin-csp`,
      options: {
        directives: {
          "script-src": `'self' 'unsafe-inline' data: www.google-analytics.com`,
          "style-src": `'self' 'unsafe-inline' fonts.googleapis.com fonts.gstatic.com`,
          "img-src": `'self' data: www.google-analytics.com`,
        },
      },
    },
  ],
};

Let's deploy again... But now I'm confused:

It says some inlines are disallowed, but they are in my directive!

Time to read the documentation

After going back to the plugin documentation, I can see they have 2 options mergeScriptHashes and mergeStyleHashes, but I didn't really understand what they are for. At the bottom of the plugin's readme, there is a link to a blog post that I should have read first.

What happens? The plugin tries its best to generate a list of the hashes for the allowed inlines script and styles that Gatsby adds. It turns out that the CSP specification states that a policy cannot use both hashes and 'unsafe-inline', and to be fair, that's what my browser is telling me in the first warning at the top of my console:

Content Security Policy: Ignoring "'unsafe-inline'" within script-src or style-src: nonce-source or hash-source specified

Lesson learned: read the warning/errors from top to bottom, not the other way around.

Solution

These 2 options were exactly what I needed, we can't use unsafe-inline with hashes which are included by the plugin, we need to disable them.

// gatsby-config.js
module.exports = {
  plugins: [
    {
      resolve: `gatsby-plugin-csp`,
      options: {
        mergeScriptHashes: false,
        mergeStyleHashes: false,
        directives: {
          "script-src": `'self' 'unsafe-inline' data: www.google-analytics.com`,
          "style-src": `'self' 'unsafe-inline' fonts.googleapis.com fonts.gstatic.com`,
          "img-src": `'self' data: www.google-analytics.com`,
          "font-src": `'self' data: fonts.gstatic.com`,
        },
      },
    },
  ],
};

At this point, it all look good visually, but when checking security headers, I still had a B grade:

Moving policy to the HTTP header

The plugin implements CSP by using a <meta> tag, not a HTTP header. They have an issue open to support HTTP headers, but it looks like it won't be supported soon as it depends on the deployment platform used. The issue links to a comment with a solution to inject the CSP to the headers with the gatsby-plugin-netlify package. The patch is pretty self-contained so I decided to replicate it on this blog. Seem to work fine and after deploying, I finally got this A grade I was after:

Success! Press merge and got it deployed.

Final words

Thanks for the very helpful resources out there, Thom Krupa & the Bejamas organisation as well as Sebastien Dubois for the script to make the header work.

The A grade is nice, however the policy I'm ending up with is not very secure and allows a lot of thing to be executed. Each time we need to resort to unsafe-inline, we're opening up a breach for attackers. I don't have much sensitive things on this site though, so it's probably good enough for this case and this seems to be the best we can do for now. I'll keep a close eye to the improvements in Gatsby and the CSP to see things evolve.

Making Celery work nicely with Django transactions

Tue, 25 Jun 2019 00:00:00 GMT

I've been using Celery in my Django projects for about 5 years now. Along the way, I made some mistakes, learned from them and picked up a few best practices. I will try to share a few of them here.

Avoid serialising complex objects

This is a pretty simple best practice, but was easy to miss as the default serialization was pickle in before v4 of Celery, the default is now JSON and enforces this. Failing that may cause hard to catch issues like when the pickle protocol changes. This happened when we migrated from Python 2 to 3 for instance. It's pretty much impossible to catch by your test suite, but will happen for sure in production.

Worker throwing `ModelInstanceDoesNotExist`

In a Django project, a complex object is very likely to be a model instance, that's where your data lives after all. The usual way to work with the previous best practice is to pass the id of a model instance when submitting the task and fetch the instance again from DB when handling the task on the worker.

# Send task as
your_task.delay(user_id=user.id)

# and run as
@app.task()
def your_task(user_id):
    user = User.objects.get(
        id=user_id
    )
    ...

However, when doing this in a naive way, this can lead to exceptions due to missing instances. It took me a while to get this one the first time it happened to me, especially given that the instance with the offending id was always there when checking manually, and there was no issues in my unit tests.

Experienced readers probably know the answer: transactions. Basically, the task is picked up by the worker before the main transaction is committed in the database. We need to delay task submission until the transaction is committed. Luckily, Django provides a hook just for that type of things: transaction.on_commit(). This is well documented in the Celery and the since 1.10 even Django uses Celery as an example in their documentation.

Problems with this solution

This hook is great, but causes a bit of boilerplate. Each time you want to use that, you need to define a lambda when you call the task. It can get quite repetitive when you start to have many tasks. Not only its longer to write and hurts readability, it may also cause some weird bugs depending on how it's used. Here is a very simplified example:

# tasks.py
@app.task()
def print_value(val):
    print(val)


# views.py
def my_view(request):
    for index in range(5):
        transaction.on_commit(
            lambda: print_value.delay(index)
        )

What would this print on the worker process? The answer is 5 times the number '4'. If you found it, well done! It's a documented behaviour of lambdas, but in a more realistic code example with longer functions and more complex parameters, it would easily be missed.

A better API

import { Callout } from "../../../components/callout";

<Callout level="note"> As of Celery v5.4, released 17th April 2024, this functionality has now been released upstream. See the Celery docs for more information. </Callout>

One idea that may come to mind is to wrap this in a decorator. This was suggested as an improvement to Django but was rejected after some discussion. I agree with the counter points: a decorator would hurt the readability of the code calling the decorated function.

What if we could wrap this boilerplate into a helper function to handle this nicely? Or better, attach this helper to our tasks? Ideally, we want to be able to call our tasks like this:

print_value.delay_on_commit(5)

That's what I'm talking about. A simple API that is easy to use, and clear to read. This also addresses the drawbacks of the previous decorator idea. The good news is that Celery provides a proper way to extend the default behaviour making it not very hard to implement.

Under the hood, the @task decorator run your task body in the run() method of the Celery base task class. This class is where the delay() callable comes from. It's possible to override the base class either when defining the task function or more globally, when defining the Celery app:

from celery import Celery


app = Celery(
    'tasks',
    task_cls='your_task:BetterTask',
)

And your base task would look like this:

from celery import Task
from django.db import transaction


class BetterTask(Task):

    def delay_on_commit(self, *args, **kwargs):
        return transaction.on_commit(
            lambda: self.delay(*args, **kwargs)
        )

And that's all you need! Now you have a nicer API, less error prone, you just need to remember to use it. How would you use it? If we modify our previous view, we would get the following:


def my_view(request):
    for index in range(5):
        print_value.delay_on_commit(index)

If we run it, we can confirm that it works now as intended, printing the numbers 1 to 5. Nice!

Closing words

As Django developers, we are generally less used to the asynchronous way of thinking than in other programming languages, as Django does not requires us to think about that (yet). On the other hand, Celery is asynchronous by its very nature, and it's easy to cause some unusual bugs in your application, if not careful. By adding a wrapper around a common pattern, to expose a better API, we can make our code easier to write, read and avoid bugs.

Static vs. Media files in Django

Tue, 18 Jun 2019 00:00:00 GMT

Django is a great Web framework for building Web applications in Python. It comes with a lot of batteries included that you'll most likely need at some point in your project. Two of them took me a while to differentiate when I started: static and media files.

After helping some less experienced people, I feel like I'm not the only one running into the confusion, so I'm hoping to clarify their differences in this post.

TL;DR

Static files are part of your application code, while media is for generated content by your application or users of your application.

Static files

Static files are managed by the staticfiles app which you need to install. It's made of several building blocks, the 3 most important ones being:

Storage classes
Templates tags
collectstatic admin command

These components work together to serve the assets in a more or less optimised way, depending on the environment. This can be altered using the following settings:

STATIC_ROOT
STATIC_URL
STATICFILES_DIRS
STATICFILES_STORAGE

Static files are usually either part of your code, or part of your dependencies' code. They can come from various places, each app may provide its own files. They are typically kept in source control. The Django admin ships with some javascript and CSS, for example, that are stored in Django's Github repository.

Local development setup

In development, the setup for static files is inefficient and optimised for convenience. It’s based on a view that, by default, looks into all the installed apps to find static files. Works great for local development, but not ideal for production.

Production setup

In production, finding files is done ahead of time via the collectstatic admin command, which you should run as part of your deployment. At a high level, the command does a very similar job as the development view, but the main difference is that it runs outside of the request-response cycle, without blocking the person visiting your website. It copies all the static files into a single location, being another folder or somewhere on another machine, which could be in a different part of the world.

My go-to solution for this used to be in django-storages. It ships with a storage class for saving your files into AWS S3. I would also add a CDN in front to help caching the assets closer to my users. The setup looked like this:

However, recently I've switched to Whitenoise, which leads to a simpler setup, and remove the need for a S3 bucket, all hosting comes from the Django app. The CDN should serve most of the traffic anyway:

I still use django-storages but only for media files, which brings us to the next section...

Media files

Media files are usually for files which are uploaded by users or generated by your application during the life of your Django project. They are typically not stored in source control.

This could be in the admin, the profile picture the user set in thei profile, some more private documents or a PDF receipt for an order that your app generate.

Basically, anything which is going into a FileField, ImageField or the likes is classified as media storage.

By default, files are stored on the local file system, and again a few settings are here to configure the behaviour:

MEDIA_ROOT
MEDIA_URL
DEFAULT_FILE_STORAGE

In some production environments -if you run your app on Heroku for instance- this default setup is not suitable. Heroku has an ephemeral file system meaning that it's cleared each time the application is updated. This is where django-storages is still useful and needed.

If your app is hosting a mix of public and private medias, I recommend to define separate storage classes for each, you can tell in the field instantiation which storage class to use:

class User:
    name = models.CharField()
    # public file
    profile_picture = models.ImageField(
        storage=PublicStorage()
    )
    # private file
    dbs_check_document = models.FileField(
        storage=PrivateStorage()
    )

The django-storages library has several backends for various hosting providers with a lot of options to customise them, so I won't go into the details of each, but here is an example of what it could look like for AWS:

from storages.backends.s3boto3 import S3Boto3Storage


class PublicStorage(S3Boto3Storage):
    default_acl = 'public'
    file_overwrite = False
    bucket_name = 'my-public-bucket'


class PrivateStorage(S3Boto3Storage):
    default_acl = 'private'
    file_overwrite = False
    bucket_name = 'my-private-bucket'

Use different class locally and in production

There is one problem with the above approach, though: by defining your storage class on the model field, that means it would potentially be shared between you local environment and production, which is probably not what you want. To solve this, we can use a factory function to either use the local file system or a remote location, depending on a setting:

def get_storage(storage_type):
    if settings.USE_REMOTE_STORAGE:
        storage_class = {
            'private': PrivateStorage,
            'public': PublicStorage,
        }[storage_type]
        return storage_class()
    return FileSystemStorage()

Which would change your model definition to:

class User:
    name = models.CharField()
    # public file
    profile_picture = models.ImageField(
        storage=get_storage('public')
    )
    # private file
    dbs_check_document = models.FileField(
        storage=get_storage('private')
    )

In a real world case, this helper should probably handle KeyError in the lookup and return the DEFAULT_FILE_STORAGE. You could also invent custom settings as you need.

Security concerns

If you decide to use django-storages for both Static and Media files, it's important that you make the separation very clear, a user of your website shouldn't be able to override one of your static assets by uploading a resource! I like to use a separate S3 Bucket for each type of storage. AWS has many options to customise the privacy of buckets and the objects their contain, and other providers have equivalent features.

Final note

While the Media storage might not be used by your application, the static files are a more essential, and any reasonable Django site will need it (at least for the admin).

I hope this post will help reduce the confusion I see frequently with beginners.

Add cache-control header to an entire S3 Bucket using Boto3

Wed, 01 May 2019 00:00:00 GMT

I recently came across a task which seem pretty generic, but for which I couldn't find an existing solution online: update the Cache-Control header for an entire S3 bucket using boto3.

Existing solutions

Do it through the AWS management console, but I wanted to script it.
Do it using Boto 2, but this is no longer installed on our system, I didn't want to reintroduce an outdated dependency.

Boto3 solution

After a bit of fiddling and digging through the documentation, I came up with this pretty simple script:

s3 = boto3.resource('s3')
bucket = s3.Bucket('my-public-bucket')
for summary in bucket.objects.filter(Prefix='static'):
    obj = summary.Object()
    obj.copy_from(
        CopySource={
            'Bucket': 'my-public-bucket',
            'Key': obj.key,
        },
        CacheControl='public,max-age=604800,immutable',
        MetadataDirective='REPLACE',
    )

The value depends a lot on your use case, you might want ot read the excellent Cache-Control for civilians post which goes into detail.

This solution uses the copy_from API, which a bit of an unnatural API for achieving this goal, but it does the trick, and seems to be the only way at the moment. Hopefully this may help others.

Failures to get there:

I initially tried via the put API, but this actually overrides the existing file with an empty one, which is not what I wanted.
The copy to itself operation failed without MetadataDirective='REPLACE'. I was about to give up but then discovered about it.

How I made 1000's of websites more secure with one line of code

Wed, 24 Apr 2019 00:00:00 GMT

Or at least a lot of them...

Learning about a new security header

Last week, I attended the London Django meetup. Among a very interesting lineup of talks, there was a lighting presentation from Adam Johnson about security headers and how Django helps with them. If you don't know what security headers are, I urge you to watch the talk, it's a bit about Django but it applies about the web in general.

While I knew the ones that comes with Django and the very important Content-Security-Policy, I discovered the Referrer-Policy header, including the story of the Referer header mispelling from the early days of the web.

In the presentation, Adam mentioned a tool to check your site and educate about all the possible security headers: Scott Helme's securityheaders.com. I was already following Scott on Twitter -I learned a great deal about CSP thanks to him- but although he probably mentioned this tool, I forgot about it.

Fixing my sites

After the presentation, I decided to check for a couple of my recent sites I built with Gatsby which I expected to be pretty well covered. After checking, they were pretty well covered, but to my surprise, the Referrer-Polocy wasn't set.

I quickly found that this could be easily set via an option from gatsby-plugin-netlify. I started adding it and set it to the suggested value of 'same-origin', which seems pretty sensible:

module.exports = {
  ...
  plugins: [
    ...
    {
      resolve: `gatsby-plugin-netlify`,
      options: {
        headers: {
          '/*': ['Referrer-Policy: same-origin'],
        },
      },
    },
  ],
}

It all worked fine so I was going to add it to my other website, but then it struck me that maybe I can push this to all websites built with Gatsby by contributing upstream, & multiply my impact! A quick check for gatsbyjs.org showed similar results as my own sites:

Fixing 1000's of website at once

So I went ahead and forked the repository on Github and quickly located where the Netlify plug-in is located in their monorepo architecture. Once in there, the plug-in codebase is actually pretty small so it was not hard to find the entry point and where to make the change, in their SECURITY_HEADERS constants:

export const SECURITY_HEADERS = {
     `X-Frame-Options: DENY`,
     `X-XSS-Protection: 1; mode=block`,
     `X-Content-Type-Options: nosniff`,
     `Referrer-Policy: same-origin`,
   ],
 }

Ideally I wanted to write tests but couldn't find any for the existing headers. Since the change was small, I decided to open a draft pull request, with the plan to fix potential test failures later, but there was none. I marked it as ready for review and at this point I was expecting to wait for a while.

Results

After this got merged, I wanted to see my change propagated to the gatsbyjs.org site. I checked again and here it was:

Next steps

It would be nice to set the CSP, but as Adam pointed in his talk, this is a big one to implement -especially on existing sites- and there is no simple default. Gatsby has an open issue to track some of the challenges around CSP.

I'm planning to add it to my websites and see how this goes, I'll probably use gatsby-plugin-csp as it looks promising.

New year, new laptop

Sun, 14 Jan 2018 00:00:00 GMT

A couple of days ago, I accidentally dropped my old laptop on the floor. It was an old Macbook Pro from mid 2010, which I've upgraded a couple of times by swapping a HDD to a SDD and then by doubling the RAM twice. I had 1TB storage and 16GB of memory, but the old processor was starting to age. After almost 8 years of service, its life came to an end.

After weighting the options, I decided to go for a Macbook Pro 15 inches, and while it was being delivered, I spent some time updating my setup scripts. It used to be a shell script calling Homebrew directly, but I recently stumble upon Homebrew bundle and decided to use that instead. It was a good opportunity to review what I no longer needed and what was missing from it.

When I received the new machine, I downloaded the files from there and ran the entry point bash script, and it worked great. The settings were restored using mackup, which I'm glad I've set it up recently.

However, I had a lot of environments setup in Insomnia (an alternative to Postman which I prefer), and sadly mackup didn't carry them over. It's probably not supporting that (yet?), so it might be an opportunity to improve this great project. My preferences from PyCharm were not fully backed up either, it looks like something changed recently as other people have reported a similar problem in Jetbrains products.

I also use Cerebro which is bit like Spotlight on steroids. The current Homebrew version (0.3.2) is crashing, so I had to grab the previous version and install it manually. I couldn't find how to install a specific cask version. Hopefully this issue will be resolved soon. Also, it looks like Docker for Mac is not installable via Homebrew, I had to go and get it from their website.

Lastly, there is still a chicken and egg situation where I need to login to Dropbox first, which requires the password from 1password, which settings are backed up in mackup.

I'm very satisfied overall, the setup has been almost entirely automated.

UPDATE: I've submitted a pull request to Mackup to add support for Insomnia, as it was very simple to do, and it just got merged!

UPDATE II: Since the Cerebro issue seems to be stuck, I've sent a PR to homebrew-cask to revert the 0.3.2 upgrade and install 0.3.1 instead.

Hello World!

Wed, 29 Nov 2017 00:00:00 GMT

Setting up my blog using the Jekyll static site generator and Github pages. This is using the remote theme feature and is I'm trying to get up and running with Minimal Mistakes.

So far it's mostly looking good, but there are few issues with listing the posts on the home page. I might write another post to explain in details.