Manage deprecations with Python warnings in a Django project

Python has a warnings module in its standard library that can be very helpful tool to manage deprecations. It’s commonly used by libraries but can also be useful to manage internal changes on big projects with lots of concurrent changes.

The library use case

Libraries with lots of users sometimes want to notify their users of upcoming changes that they plan break in the future. On the receiving end, users can control which action to take, and this action may be different depending on the context.

For instance, Django defines (at time of writing) a few warnings RemovedInDjango60Warning, RemovedInDjango61Warning, aliased to respectively RemovedInNextVersionWarning and RemovedAfterNextVersionWarning. While it may be useful to see these warnings in local development or while running tests on CI, it could be noisy to have them on production, potentially distracting us from other important signals.

Warnings can be filtered in various ways:

• with the -W command line argument when running Python (python -W manage.py ...)
• with the PYTHONWARNINGS environment variable
• programmatically, in your Python code with warnings.filterwarnings(...)
• in pytest, using the -W option or via the filterwarnings config.

I usually try to enable all warnings during my CI runs, and not show them anywhere else. If there is too much noise in my CI, then it’s a sign that I should fix the warnings. If you need some more advanced filtering, the filtering syntax is quite flexible, the pytest documentation does a good job at giving a few examples of filters.

Using warnings internally

Warnings can also be useful to manage deprecations within the bound of an application, without external downstream consumers of the code, but with lots of developers. In big project, it’s quite common to have a shared internal library of utilities to make things easier to do, or just have service layers that other parts of the system use. After a while, some of these parts can end up being used 100s or 1000s of times.

When we inevitably reach the limit of a given system and want to refactor it towards a better approach, it can take time to transition. In some cases, data might need to be migrated over in the middle, and should to be released in stages to avoid downtime. While the 2 or 3 stages are implemented, the rest of team keeps moving at full speed, and may keep using the old pattern instead of using the new way of doing things.

Emitting a custom warning is a good way to manage these gradual deprecations. It’s much easier to control what happens on CI that it is on production, and you can opt to treat your warning as an error in this context, hence breaking the tests and signaling others towards the new way of doing things.

Example

Imagine you have a is_user_allowed function in an internal library that you want to deprecate in favour of a new_is_user_allowed. This function may be used in lots of places, to do higher level operations. You could add a call to emit a warning in your deprecated function:

# consumer.py
from lib import is_user_allowed

def do_high_level_stuff(user):
    is_user_allowed(user)
    ...

# lib.py
import warnings

def is_user_allowed(user):
    warnings.warn(
        "is_user_allowed is deprecated",
        category=DeprecationWarning,
        stacklevel=2,
    )
    return new_is_user_allowed(user)

It’s a good idea to pass a stacklevel of at least 2, otherwise the warning will appear to come from lib.py. By passing a stacklevel of 2, the warning will appear to come from consumer.py, hence making it much easier to locate usages.

You can configure pytest to treat this warning as error, except in existing places where it’s already used:

# pyproject.toml
[tool.pytest.ini_options]
filterwarnings = [
    "once",
    "error:is_user_allowed is deprecated:DeprecationWarning",
    "once:is_user_allowed is deprecated:DeprecationWarning:consumer",
    "...",
]

If other teammates working on other branches start using it in new places, their test will start failing, preventing them to introduce new usages of the deprecated function. This first step can be released while you gradually fix all the existing usages, removing them from the pytest’s filterwarnings list as you go.

You may want to go one step further and log usages on production for a little while. Warnings aren’t integrated with the standard logging module by default, but it’s not very hard to setup. Here is how you can modify your Django LOGGING setting to do it:

# settings.py
import logging

logging.captureWarnings(True)LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
        },
    },
    "root": {
        "handlers": ["console"],
        "level": "WARNING",
    },
    "loggers": {
        "apps": {
            "handlers": ["console"],
            "level": "INFO",
            "propagate": False,
        },
        "py.warnings": {            "handlers": ["console"],            "level": "WARNING",            "propagate": False,        },    },
}

Integration is enabled with logging.captureWarnings(True) and use the py.warnings logger, hence the last logger definition. Here are the official docs on how to integrate that, if you’re interested to read more.

The final piece to get our warning actually emitted, is to set the filter via the PYTHONWARNINGS environment variable:

PYTHONWARNINGS="all:is_user_allowed is deprecated:DeprecationWarning:"

If you need to set multiple filters in the env variable, they need be comma separated:

PYTHONWARNINGS="all:is_user_allowed is deprecated:DeprecationWarning:,all:old_function is also deprecated:DeprecationWarning:"

Conclusion

Hope this post will help you get up to speed with warnings management, the last bit to integrate with the logging module was new to me.