Structuring a Python Project with PDM

Published on Nov 12, 2023

Tutorial

Python

#Python

#PDM

This tutorial shows you how to structure a Python project with PDM, a modern Python package and dependency manager supporting the latest PEP standards.

Background

PDM is a modern Python package and dependency manager supporting the latest PEP standards, including PEP 621 project metadata and a PEP 517 build backend. PDM simplifies the process od depenedency management and lifecycle management of Python projects.

Alternatives:

Objectives

This tutorial covers the following steps:

Initialize a new project
Configure development environment
Manage development dependencies
Build and Publish

Prerequisites

Python 3.7+
PDM

Install PDM

Install PDM with `pipx`

pipx is a tool to install and run applications in isolated virtual environments. It is remommended to install Python application in an isolated environemnt to acoid potential dependency conflicts.

Install pipx and ensure directories necessary for pipx operation are in PATH environment variable:

python -m pip install --user pipx
python -m pipx ensurepath

Install pdm with pipx:

pipx install pdm[all]

Install PDM with `pip`

You can still install pdm with pip normally:

pip install --user pdm

(Optional) Enable central installation caches of PDM

Similar to pnpm, pdm also supports central installation caches to avoid waste of disc space.

Enable the central installation caches of PDM:

pdm config install.cache on

Initialize a new project

Create a new project with pdm init:

mkdir my-project && cd my-project
pdm init

The outcome is similar to the following:

Creating a pyproject.toml for PDM...
Please enter the Python interpreter to use
0. /usr/local/bin/python (3.11)
1. /usr/local/bin/python3.12 (3.12)
2. /usr/local/bin/python3.11 (3.11)
3. /usr/local/bin/python3.10 (3.10)
Please select (0):
Would you like to create a virtualenv with /usr/local/bin/python? [y/n] (y):
Virtualenv is created successfully at /Users/rudeigerc/Developer/my-project/.venv
Is the project a library that is installable?
If yes, we will need to ask a few more questions to include the project name and build backend [y/n] (n): y
Project name (my-project):
Project version (0.1.0):
Project description ():
Which build backend to use?
0. pdm-backend
1. setuptools
2. flit-core
3. hatchling
Please select (0):
License(SPDX name) (MIT):
Author name (rudeigerc):
Author email ([email protected]):
Python requires('*' to allow any) (>=3.11):
Project is initialized successfully

Make sure that you have selected the proper Python interpreter.

Work with virtual environment

PDM will ask whether to use virtual environment when initializing a new project. It is recommended to use virtual environment to isolate the project from the global environment.

Actiavte the virtual environment in the project:

eval $(pdm venv activate)

Please refer to the documentation to learn more about how to manage virtual environments with PDM.

Configure the project

(Optional) Configure package indexes

To overcome the network issues, you can configure package indexes to use mirrors of PyPI.

Add tuna and sjtug as package indexes in pyproject.toml:

[[tool.pdm.source]]
url = "https://pypi.tuna.tsinghua.edu.cn/simple"
name = "tuna"

[[tool.pdm.source]]
url = "https://mirror.sjtu.edu.cn/pypi/web/simple"
name = "sjtug"

You can choose to ignore the stored package indexes and use the aforementioned package indexes only defined in pyproject.toml:

pdm config --local pypi.ignore_stored_index true

Configure development environment

Configure Visual Studio Code

Here are some recommended extensions for Visual Studio Code:

{
  "recommendations": [
    "charliermarsh.ruff",
    "editorconfig.editorconfig",
    "ms-python.mypy-type-checker",
    "ms-python.python",
    "ms-python.vscode-pylance",
    "njpwerner.autodocstring",
    "redhat.vscode-yaml",
    "tamasfe.even-better-toml"
  ]
}

You can define recommended extensions in workspace scope in .vscode/extensions.json

Configure Python language support

Configure Python language support in .vscode/settings.json:

{
  "python.analysis.autoImportCompletions": true,
  "python.analysis.extraPaths": [".venv/lib/python3.11/site-packages"],
  "python.analysis.fixAll": ["source.unusedImports"],
  "python.languageServer": "Pylance"
}

Note that extraPaths should match the path based on the interpeter version of Python in your virtual environment.

(Optional) Configure `.editorconfig`

EditorConfig is a tool that helps developers define and maintain consistent coding styles between different editors and IDEs.

Here is an example of .editorconfig for Python projects:

root = true

[*]
charset = utf-8
trim_trailing_whitespace = true
end_of_line = lf
indent_style = space
insert_final_newline = true
indent_size = 2

[*.py]
indent_size = 4

[pyproject.toml]
indent_size = 4

Manage development dependencies

In this section, the tutorial will cover the following topics:

Linting and formatting
Testing
Documentation

Linting and formatting

Configure `ruff`

ruff is an extremely fast Python linter and code formatter, written in Rust.

Since v0.1.2, ruff supports both linting and formatting with outstanding performance compared to existing tools. As a result, ruff is recommended as the default linter and formatter for Python projects in this tutorial.

Add ruff as a development dependency in group lint:

pdm add -dG lint ruff

Configure ruff in pyproject.toml:

[tool.ruff]
select = [
    "B", # flake8-bugbear
    "C4", # flake8-comprehensions
    "E", # pycodestyle - Error
    "F", # Pyflakes
    "I", # isort
    "W", # pycodestyle - Warning
    "UP", # pyupgrade
]
ignore = [
    "E501", # line-too-long
    "W191", # tab-indentation
]
include = ["**/*.py", "**/*.pyi", "**/pyproject.toml"]

[tool.ruff.pydocstyle]
convention = "google"

Configure ruff in .vscode/settings.json:

{
  "[python]": {
    "editor.codeActionsOnSave": {
      "source.fixAll": true,
      "source.organizeImports": true
    },
    "editor.defaultFormatter": "charliermarsh.ruff",
    "editor.formatOnSave": true,
    "editor.rulers": [88]
  }
}

Please refer to the documentation of ruff for more details about configuration.

Alternatives:

Linter
- PyCQA/flake8
- pylint-dev/pylint
Formatter

Configure `mypy`

mypy is an optional static type checker for Python.

pdm add -dG lint mypy

Configure mypy in pyproject.toml:

[tool.mypy]
strict = true

If you run mypy with the —strict flag, you will basically never get a type related error at runtime without a corresponding mypy error, unless you explicitly circumvent mypy somehow.

However, this flag will probably be too aggressive if you are trying to add static types to a large, existing codebase. See Using mypy with an existing codebase for suggestions on how to handle that case.

Configure mypy in .vscode/settings.json:

{
  "mypy-type-checker.path": [".venv/bin/mypy"]
}

Please refer to the documentation of mypy for more details about configuration.

Alternatives:

Configure PDM scripts

PDM supports running scripts or commands with local packages loaded, similar to npm.

[tool.pdm.scripts]
lint = "ruff ."
fmt = "ruff format ."

Please refer to the documentaion of PDM scripts for more details.

(Optional) Configure `pre-commit`

pre-commit is a framework for managing and maintaining multi-language pre-commit hooks.

Add pre-commit as a development dependency in group lint:

pdm add -dG lint pre-commit

Add a pre-commit configuration file .pre-commit-config.yaml:

repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.5.0
    hooks:
      - id: check-yaml
        args:
          - --unsafe
      - id: end-of-file-fixer
      - id: trailing-whitespace
  - repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.1.2
    hooks:
      - id: ruff
      - id: ruff-format
  - repo: https://github.com/pre-commit/mirrors-mypy
    rev: v1.6.1
    hooks:
      - id: mypy
        language: system

Install the Git hook scripts:

pdm run pre-commit install

pre-commit will be triggered automatically before each commit.

Besides, you can also modify the aforementioned pdm.scripts section in pyproject.toml:

[tool.pdm.scripts]
- lint = "ruff ."
- fmt = "ruff format ."
+ lint = "pre-commit run --all-files"

Now you can run pdm run lint to lint the project manually.

Testing

Configure `pytest`

Pytest is a testing framework for Python.

Add pytest and pytest-cov as development dependencies in group test:

pdm add -dG pytest pytest-cov

Configure testing support in .vscode/settings.json:

{
  "python.testing.pytestArgs": ["tests"],
  "python.testing.pytestEnabled": true,
  "python.testing.pytestPath": ".venv/bin/pytest",
  "python.testing.unittestEnabled": false
}

With the configuration above, you can run tests with the built-in test runner of Visual Studio Code.

Alternatives:

unittest

(Optional) Documentation

MkDocs is a static site generator for project documentation, and mkdocs-material is a powerful documentation framework on top of MkDocs.

Add mkdocs and mkdocs-material as development dependencies in group docs:

pdm add -dG docs mkdocs mkdocs-material

Configure YAML schema validation in .vscode/settings.json as mentioned in the documentation:

{
  "yaml.schemas": {
    "https://squidfunk.github.io/mkdocs-material/schema.json": "mkdocs.yml"
  },
  "yaml.customTags": [
    "!ENV scalar",
    "!ENV sequence",
    "tag:yaml.org,2002:python/name:material.extensions.emoji.to_svg",
    "tag:yaml.org,2002:python/name:material.extensions.emoji.twemoji",
    "tag:yaml.org,2002:python/name:pymdownx.superfences.fence_code_format"
  ]
}

Initialize a new documentation:

pdm run mkdocs new .

This will create the following structure:

├─ docs/
│  └─ index.md
└─ mkdocs.yml

Enable mkdocs-material in mkdocs.yml:

theme:
  name: material

Add a script to run mkdocs in pyproject.toml:

[tool.pdm.scripts]
docs = "mkdocs serve"

Start the development server:

pdm run docs

The detailed configuration of mkdocs and mkdocs-material is out of the scope of this tutorial. Please refer to the documentation for more details.

Alternatives:

Build and Publish

After configuring the project and finishing coding and testing, you can build and publish the project to PYPI.

PDM also provides a PEP 517 build backend.

[build-system]
requires = ["pdm-backend"]
build-backend = "pdm.backend"

Configure project metadata

PEP 621 is a standard for Python project metadata.

Define project metadata in pyproject.toml:

[project]
name = "my-project"
version = "0.1.0"
description = ""
authors = [{name = "rudeigerc", email="[email protected]"}]
dependencies = []
requires-python = ">=3.11"
readme = "README.md"
license = {text = "MIT"}

Package version

PDM backend supports dynamic project version.

Remove the version field and import version in the dynamic field in pyproject.toml:

[project]
- version = "0.1.0"
+ dynamic = ["version"]

From a given file path

[tool.pdm.version]
source = "file"
path = "src/my-package/__init__.py"

The file src/my-package/__init__.py should contain a __version__ variable:

__version__ = "0.1.0"

From SCM tag

[tool.pdm.version]
source = "scm"

Tag current commit with version v0.0.1:

git tag v0.0.1

Check the artifacts with pdm build:

pdm build

Configure PYPI

To upload your package to PYPI, you need to create an account on PYPI and generate an API token in Account settings > API Tokens > Add API Token.

With your username and the generated token, configure PYPI remote repository in pyproject.toml:

pdm config repository.pypi.url https://pypi.org/simple
pdm config repository.pypi.username <username>
pdm config repository.pypi.password <api-token>

Check the configuration of repository pypi:

pdm config

The outcome is similar to the following:

Home configuration (config.toml):
repository.pypi.url = "https://pypi.org/simple"
repository.pypi.username = <username>
repository.pypi.password = <api-token>

Build and publish the project to PYPI:

pdm publish -r pypi

Note that for test purpose, you should publish the project to Test PYPI:

pdm config repository.testpypi.url https://test.pypi.org/simple
pdm config repository.testpypi.username <username>
pdm config repository.testpypi.password <api-token>

pdm publish -r testpypi

Please refer the documenation of PDM Backend and Python Package User Guide for more details about publishing Python packages.

Clean up

Delete the project

cd .. && rm -rf my-project

Appendix

Libraries

ruff - An extremely fast Python linter and code formatter, written in Rust.
mypy - Optional static typing for Python
mkdocs - Project documentation with Markdown.
mkdocs-material - Documentation that simply works
pytest - The pytest framework makes it easy to write small tests, yet scales to support complex functional testing
pytest-cov - Coverage plugin for pytest.
pre-commit - A framework for managing and maintaining multi-language pre-commit hooks.

Template

Since v2.8, PDM supports initializing a project from a template.

I have created a template rudeigerc/pdm-template-rudeigerc for my personal use mainly based on this tutorial.

You can initialize a project from this template with the following command:

pdm init https://github.com/rudeigerc/pdm-template-rudeigerc

Here are some other recommended Python project templates:

microsoft/python-package-template

← Previous 排查通过 Ingress 暴露的 Jupyter Lab 的伪终端间歇性输出字符问题 Next → KubeCon + CloudNativeCon + Open Source Summit China 2023 Recap

Structuring a Python Project with PDM

Background

Objectives

Prerequisites

Install PDM

Install PDM with pipx

Install PDM with pip

(Optional) Enable central installation caches of PDM

Initialize a new project

Work with virtual environment

Configure the project

(Optional) Configure package indexes

Configure development environment

Configure Visual Studio Code

Configure Python language support

(Optional) Configure .editorconfig

Manage development dependencies

Linting and formatting

Configure ruff

Configure mypy

Configure PDM scripts

(Optional) Configure pre-commit

Testing

Configure pytest

(Optional) Documentation

Build and Publish

Configure project metadata

Package version

From a given file path

From SCM tag

Configure PYPI

Clean up

Delete the project

Appendix

Libraries

Template

Install PDM with `pipx`

Install PDM with `pip`

(Optional) Configure `.editorconfig`

Configure `ruff`

Configure `mypy`

(Optional) Configure `pre-commit`

Configure `pytest`