Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 4 additions & 5 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
name: Lint and type check

# The public repository (nauticalab/orcapod-python) has no organization-level required checks, so
# the checks CLAUDE.md lists under "Lint, format and type checks" run here. pytest is covered by
# run-tests.yml. The job does not run on the development repository, whose required CI runs the
# same commands (ITL-1007).
# The checks CLAUDE.md lists under "Lint, format and type checks", in this repository's own
# workflow; pytest is covered by run-tests.yml. They run on every repository, the development
# one included (ITL-1181): once it is public, the organization's required workflow no longer
# applies to it (ITL-1193), and these checks are the ones that remain.

on:
push:
Expand All @@ -13,7 +13,6 @@ on:
jobs:
lint:
name: Ruff and basedpyright
if: github.repository == 'nauticalab/orcapod-python'
runs-on: ubuntu-latest
timeout-minutes: 20
permissions:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ jobs:
run: uv sync --locked --all-extras --dev --python ${{ matrix.python-version }}

- name: Run tests
run: uv run --python ${{ matrix.python-version }} pytest -m "not postgres" --tb=short -q
run: uv run --python ${{ matrix.python-version }} pytest -n logical -m "not postgres" --tb=short -q

license-check:
needs: [validate-branch]
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/run-postgres-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,8 @@ jobs:
run: uv sync --locked --all-extras --dev --python ${{ matrix.python-version }}

- name: Run PostgreSQL tests
run: uv run --python ${{ matrix.python-version }} pytest -v -m postgres --postgres --cov=src --cov-report=term-missing --cov-report=xml
# Serial (-n 0): the tests share one PostgreSQL server (ITL-1181).
run: uv run --python ${{ matrix.python-version }} pytest -n 0 -v -m postgres --postgres --cov=src --cov-report=term-missing --cov-report=xml
env:
PGHOST: localhost
PGPORT: 5432
Expand Down
16 changes: 13 additions & 3 deletions .github/workflows/run-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,15 @@ jobs:
needs: license-check
runs-on: ubuntu-latest
strategy:
# A pull request runs one job, on Python 3.13, whose coverage is the one reported; a
# push (main) runs all three versions (developer, November 16, 2026; ITL-1181). A
# failure only Python 3.11 or 3.12 shows is first seen on main.
matrix:
python-version: ["3.11", "3.12", "3.13"]
python-version: ${{ fromJSON(github.event_name == 'pull_request' && '["3.13"]' || '["3.11", "3.12", "3.13"]') }}
env:
# sys.monitoring (Python 3.12+) measures coverage at a fraction of the default
# tracer's cost; Python 3.11 has no sys.monitoring and keeps the C tracer.
COVERAGE_CORE: ${{ matrix.python-version == '3.11' && 'ctrace' || 'sysmon' }}

steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
Expand All @@ -39,7 +46,9 @@ jobs:
run: uv run --python ${{ matrix.python-version }} python scripts/check_golden_tags.py

- name: Run tests
run: uv run --python ${{ matrix.python-version }} pytest -v -rsxX --durations=20 -m "not postgres" --cov=src --cov-report=term-missing --cov-report=xml
# In parallel on every core of the runner (pytest-xdist; ITL-1181); pytest-cov
# combines the workers' coverage into one report.
run: uv run --python ${{ matrix.python-version }} pytest -n logical -v -rsxX --durations=20 -m "not postgres" --cov=src --cov-report=term-missing --cov-report=xml

- name: Upload coverage reports to Codecov
if: github.repository == 'metamorphic-labs/orcapod-python'
Expand Down Expand Up @@ -128,7 +137,8 @@ jobs:
env:
SPIRAL_CLIENT_ID: ${{ secrets.SPIRAL_CLIENT_ID }}
SPIRAL_PRIVATE_KEY: ${{ secrets.SPIRAL_PRIVATE_KEY }}
run: uv run --python "3.11" pytest tests/test_databases/test_spiraldb_connector_integration.py -v
# Serial (-n 0): the integration tests share one SpiralDB project (ITL-1181).
run: uv run --python "3.11" pytest -n 0 tests/test_databases/test_spiraldb_connector_integration.py -v

license-check:
uses: ./.github/workflows/_license-check.yml
Expand Down
26 changes: 23 additions & 3 deletions .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -247,18 +247,38 @@ Developer decisions, November 12, 2026 (ITL-1007); the full procedure is RELEASI
Release workflow, or touches a setting, secret or environment there. The push happens only on
the developer's explicit go. The v0.2.0 tag is never pushed.
- Linear, Codecov and SpiralDB jobs run only when
github.repository == 'metamorphic-labs/orcapod-python'; lint.yml runs only on nauticalab.
github.repository == 'metamorphic-labs/orcapod-python'; lint.yml runs on both repositories (ITL-1181).

## Lint, format and type checks

Every pull request must pass the metamorphic-labs required CI, which runs exactly these
commands (pytest on Python 3.11, 3.12 and 3.13; the others on 3.11):
Every pull request must pass these checks; CI runs exactly these commands:

uv run --with ruff ruff check .
uv run --with ruff ruff format --check .
uv run --with basedpyright basedpyright
uv run pytest tests/

What runs where (developer, November 16, 2026; ITL-1181):

- The metamorphic-labs required CI (organization ruleset `python-required-ci`, a shared
workflow this repository cannot change) runs all four on every pull request: pytest on
Python 3.11, 3.12 and 3.13, the others on 3.11. It applies only while the repository is
private (ITL-1193).
- `lint.yml` (job `Ruff and basedpyright`) runs the first three on every pull request and
push to `main`, on both repositories. `run-tests.yml` runs pytest with coverage: on a pull
request one job, `test (3.13)`, whose coverage is the one reported to Codecov; on a push
to `main`, Python 3.11, 3.12 and 3.13. Once the required CI no longer applies, a pull
request is tested on Python 3.13 only, and a failure only 3.11 or 3.12 shows is first
seen on `main`.
- In CI the tests run in parallel, one worker per logical CPU (pytest-xdist, `-n logical`;
`-n auto` counts physical cores, one on a GitHub runner). One place turns it on,
`tests/conftest.py`, when `CI` is `true` and no `-n` is given; `-n 0` keeps a run serial
(the PostgreSQL and SpiralDB integration jobs). Coverage on Python 3.12 and later uses
`COVERAGE_CORE=sysmon`.
- Locally the tests run serially. `uv run pytest -n 4 tests/` runs them on four workers;
each worker holds its own copy of the suite's memory, so do it only on a machine with
cores and memory to spare, never beside another heavy process.

- New and changed code must be ruff-clean and ruff-formatted, and must add no
basedpyright errors. Warnings are reported but do not fail the check
(`failOnWarnings = false`); still do not add them needlessly. Run
Expand Down
26 changes: 23 additions & 3 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -256,12 +256,11 @@ Developer decisions, November 12, 2026 (ITL-1007); the full procedure is `RELEAS
Release workflow, or touches a setting, secret or environment there. The push happens only on
the developer's explicit go. The `v0.2.0` tag is never pushed.
- Linear, Codecov and SpiralDB jobs run only when
`github.repository == 'metamorphic-labs/orcapod-python'`; `lint.yml` runs only on `nauticalab`.
`github.repository == 'metamorphic-labs/orcapod-python'`; `lint.yml` runs on both repositories (ITL-1181).

## Lint, format and type checks

Every pull request must pass the metamorphic-labs required CI, which runs exactly these
commands (pytest on Python 3.11, 3.12 and 3.13; the others on 3.11):
Every pull request must pass these checks; CI runs exactly these commands:

```
uv run --with ruff ruff check .
Expand All @@ -270,6 +269,27 @@ uv run --with basedpyright basedpyright
uv run pytest tests/
```

What runs where (developer, November 16, 2026; ITL-1181):

- The metamorphic-labs required CI (organization ruleset `python-required-ci`, a shared
workflow this repository cannot change) runs all four on every pull request: pytest on
Python 3.11, 3.12 and 3.13, the others on 3.11. It applies only while the repository is
private (ITL-1193).
- `lint.yml` (job `Ruff and basedpyright`) runs the first three on every pull request and
push to `main`, on both repositories. `run-tests.yml` runs pytest with coverage: on a pull
request one job, `test (3.13)`, whose coverage is the one reported to Codecov; on a push
to `main`, Python 3.11, 3.12 and 3.13. Once the required CI no longer applies, a pull
request is tested on Python 3.13 only, and a failure only 3.11 or 3.12 shows is first
seen on `main`.
- In CI the tests run in parallel, one worker per logical CPU (pytest-xdist, `-n logical`;
`-n auto` counts physical cores, one on a GitHub runner). One place turns it on,
`tests/conftest.py`, when `CI` is `true` and no `-n` is given; `-n 0` keeps a run serial
(the PostgreSQL and SpiralDB integration jobs). Coverage on Python 3.12 and later uses
`COVERAGE_CORE=sysmon`.
- Locally the tests run serially. `uv run pytest -n 4 tests/` runs them on four workers;
each worker holds its own copy of the suite's memory, so do it only on a machine with
cores and memory to spare, never beside another heavy process.

- New and changed code must be ruff-clean and ruff-formatted, and must add no
basedpyright errors. Warnings are reported but do not fail the check
(`failOnWarnings = false`); still do not add them needlessly. Run
Expand Down
6 changes: 5 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@ The project uses [uv](https://docs.astral.sh/uv/):
uv sync --all-extras --dev
```

Every pull request must pass these checks (pytest on Python 3.11, 3.12 and 3.13):
Every pull request must pass these checks:

```bash
uv run --with ruff ruff check .
Expand All @@ -151,6 +151,10 @@ uv run --with basedpyright basedpyright
uv run pytest tests/
```

CI runs the tests in parallel (pytest-xdist). Locally they run serially; `uv run pytest -n 4 tests/`
runs them on four workers, each holding its own copy of the suite's memory, so use it only on a
machine with cores and memory to spare.

The PostgreSQL integration tests are skipped unless asked for
(`uv run pytest -m postgres --postgres`) and need a running PostgreSQL server, reached
through the standard `PG*` environment variables; see
Expand Down
17 changes: 17 additions & 0 deletions changelog.d/ITL-1181.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
### Changed

- **Faster CI rounds** — CI runs the tests in parallel on every core of the runner
(pytest-xdist, a new development dependency), turned on by `tests/conftest.py` when `CI` is
`true` and no `-n` is given; local runs stay serial (`uv run pytest -n 4 tests/` runs them
in parallel). A pull request runs the coverage job (`run-tests.yml`) once, on Python 3.13
with `COVERAGE_CORE=sysmon`; a push to `main` runs it on Python 3.11, 3.12 and 3.13. The
lint and type check workflow (`lint.yml`) now runs on the development repository too. The
Release workflow's tests run in parallel. (ITL-1181)

### Fixed

- **Interrupt tests in a background process** — the tests in
`tests/test_pipeline/test_job_run_in_event_loop.py` that press the notebook's interrupt
button (`_thread.interrupt_main`) install Python's default SIGINT handler for their
duration, so they pass in a process that ignores SIGINT (one started in the background).
(ITL-1181)
64 changes: 64 additions & 0 deletions docs/dev/scripts/itl_1181_per_file_times.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
"""Sum test time per test file from pytest JUnit XML reports (ITL-1181).

The organization's required pytest check uploads one report per Python version
(``test-results-py3.X/test-results.xml``); download them with
``gh run download <run id>`` and pass their paths. Prints the files by total time over
all reports, with each file's share and the running share, leaving out the files that
protect what CI must always check (golden hashes, the hash label rule, the documentation
samples and build, the changelog and design-issue checks, the hash site inventory).

uv run python docs/dev/scripts/itl_1181_per_file_times.py test-results-py3.*/test-results.xml
"""

from __future__ import annotations

import collections
import sys
import xml.etree.ElementTree as ET

PROTECTED = (
"tests/test_hashing/golden/",
"test_hash_label_rule.py",
"test_hash_site_inventory.py",
"test_doc_samples.py",
"test_mkdocs_build.py",
"test_changelog_fragments.py",
"test_design_issues.py",
)


def module_path(classname: str) -> str:
"""The test file of a JUnit ``classname`` (``tests.a.test_b.TestC`` to ``tests/a/test_b.py``)."""
parts = classname.split(".")
while parts and parts[-1][:1].isupper():
_ = parts.pop()
return "/".join(parts) + ".py"


def main(paths: list[str], top: int = 19) -> None:
"""Print the ``top`` slowest unprotected files over the reports at ``paths``."""
seconds: collections.defaultdict[str, float] = collections.defaultdict(float)
total = 0.0
for path in paths:
for case in ET.parse(path).getroot().iter("testcase"):
duration = float(case.get("time") or 0)
seconds[module_path(case.get("classname", ""))] += duration
total += duration
print(f"{len(paths)} report(s), {total / len(paths):.0f} s of test time per report")
running = 0.0
shown = 0
for name, spent in sorted(seconds.items(), key=lambda item: -item[1]):
if any(marker in name for marker in PROTECTED):
continue
running += spent
shown += 1
print(
f"{shown:2d} {spent / len(paths):6.1f} s {100 * spent / total:4.1f} %"
+ f" (running {100 * running / total:5.1f} %) {name}"
)
if shown == top:
break


if __name__ == "__main__":
main(sys.argv[1:])
Loading
Loading