From dcf0ba372ff7cc955b6f7cf994ab7104b6db6f90 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Mon, 16 Nov 2026 22:31:42 -0800 Subject: [PATCH 1/3] ci(tests): run CI tests in parallel, one coverage job per pull request (ITL-1181) - pytest-xdist as a dev dependency; tests/conftest.py turns on -n auto when CI is true and no -n is given (never on an xdist worker), so the organization's required pytest check runs in parallel too. Local runs stay serial. - run-tests.yml: a pull request runs one job, test (3.13), with COVERAGE_CORE=sysmon, reporting coverage; a push runs 3.11, 3.12 and 3.13. The PostgreSQL and SpiralDB integration jobs stay serial (-n 0). - lint.yml runs on the development repository too (ITL-1193 makes it public). - release.yml: the test step runs in parallel. - The interrupt tests in test_job_run_in_event_loop.py install the default SIGINT handler for their duration, so they pass in a process that ignores SIGINT. - Temporary measurement jobs (serial coverage baseline, 3.11 cost), removed before merge. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/lint.yml | 9 ++-- .github/workflows/release.yml | 2 +- .github/workflows/run-postgres-tests.yml | 3 +- .github/workflows/run-tests.yml | 46 +++++++++++++++++-- .zed/rules | 25 ++++++++-- CLAUDE.md | 25 ++++++++-- README.md | 6 ++- changelog.d/ITL-1181.md | 17 +++++++ pyproject.toml | 1 + tests/conftest.py | 31 +++++++++++++ .../test_job_run_in_event_loop.py | 22 ++++++++- uv.lock | 24 ++++++++++ 12 files changed, 193 insertions(+), 18 deletions(-) create mode 100644 changelog.d/ITL-1181.md diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 71a307647..589c490ab 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -1,9 +1,9 @@ name: Lint and type check -# The public repository (nauticalab/orcapod-python) has no organization-level required checks, so -# the checks CLAUDE.md lists under "Lint, format and type checks" run here. pytest is covered by -# run-tests.yml. The job does not run on the development repository, whose required CI runs the -# same commands (ITL-1007). +# The checks CLAUDE.md lists under "Lint, format and type checks", in this repository's own +# workflow; pytest is covered by run-tests.yml. They run on every repository, the development +# one included (ITL-1181): once it is public, the organization's required workflow no longer +# applies to it (ITL-1193), and these checks are the ones that remain. on: push: @@ -13,7 +13,6 @@ on: jobs: lint: name: Ruff and basedpyright - if: github.repository == 'nauticalab/orcapod-python' runs-on: ubuntu-latest timeout-minutes: 20 permissions: diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index ed88ab0b5..7111d89f3 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -66,7 +66,7 @@ jobs: run: uv sync --locked --all-extras --dev --python ${{ matrix.python-version }} - name: Run tests - run: uv run --python ${{ matrix.python-version }} pytest -m "not postgres" --tb=short -q + run: uv run --python ${{ matrix.python-version }} pytest -n auto -m "not postgres" --tb=short -q license-check: needs: [validate-branch] diff --git a/.github/workflows/run-postgres-tests.yml b/.github/workflows/run-postgres-tests.yml index 83acccdb6..e1e068a8e 100644 --- a/.github/workflows/run-postgres-tests.yml +++ b/.github/workflows/run-postgres-tests.yml @@ -46,7 +46,8 @@ jobs: run: uv sync --locked --all-extras --dev --python ${{ matrix.python-version }} - name: Run PostgreSQL tests - run: uv run --python ${{ matrix.python-version }} pytest -v -m postgres --postgres --cov=src --cov-report=term-missing --cov-report=xml + # Serial (-n 0): the tests share one PostgreSQL server (ITL-1181). + run: uv run --python ${{ matrix.python-version }} pytest -n 0 -v -m postgres --postgres --cov=src --cov-report=term-missing --cov-report=xml env: PGHOST: localhost PGPORT: 5432 diff --git a/.github/workflows/run-tests.yml b/.github/workflows/run-tests.yml index f5083fe73..1dda9fad8 100644 --- a/.github/workflows/run-tests.yml +++ b/.github/workflows/run-tests.yml @@ -11,8 +11,15 @@ jobs: needs: license-check runs-on: ubuntu-latest strategy: + # A pull request runs one job, on Python 3.13, whose coverage is the one reported; a + # push (main) runs all three versions (developer, November 16, 2026; ITL-1181). A + # failure only Python 3.11 or 3.12 shows is first seen on main. matrix: - python-version: ["3.11", "3.12", "3.13"] + python-version: ${{ fromJSON(github.event_name == 'pull_request' && '["3.13"]' || '["3.11", "3.12", "3.13"]') }} + env: + # sys.monitoring (Python 3.12+) measures coverage at a fraction of the default + # tracer's cost; Python 3.11 has no sys.monitoring and keeps the C tracer. + COVERAGE_CORE: ${{ matrix.python-version == '3.11' && 'ctrace' || 'sysmon' }} steps: - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 @@ -39,7 +46,9 @@ jobs: run: uv run --python ${{ matrix.python-version }} python scripts/check_golden_tags.py - name: Run tests - run: uv run --python ${{ matrix.python-version }} pytest -v -rsxX --durations=20 -m "not postgres" --cov=src --cov-report=term-missing --cov-report=xml + # In parallel on every core of the runner (pytest-xdist; ITL-1181); pytest-cov + # combines the workers' coverage into one report. + run: uv run --python ${{ matrix.python-version }} pytest -n auto -v -rsxX --durations=20 -m "not postgres" --cov=src --cov-report=term-missing --cov-report=xml - name: Upload coverage reports to Codecov if: github.repository == 'metamorphic-labs/orcapod-python' @@ -128,7 +137,38 @@ jobs: env: SPIRAL_CLIENT_ID: ${{ secrets.SPIRAL_CLIENT_ID }} SPIRAL_PRIVATE_KEY: ${{ secrets.SPIRAL_PRIVATE_KEY }} - run: uv run --python "3.11" pytest tests/test_databases/test_spiraldb_connector_integration.py -v + # Serial (-n 0): the integration tests share one SpiralDB project (ITL-1181). + run: uv run --python "3.11" pytest -n 0 tests/test_databases/test_spiraldb_connector_integration.py -v + + # TEMPORARY (ITL-1181): measurements for the pull request; removed before it merges. + # serial-ctrace is today's configuration (serial, default tracer), whose coverage total the + # parallel test (3.13) job's must equal; xdist-ctrace is the cost of a pull-request job on + # Python 3.11 instead of 3.13. + measure: + name: measure (${{ matrix.label }}) + if: github.event_name == 'pull_request' + needs: license-check + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + include: + - { label: "3.13 serial ctrace", python-version: "3.13", workers: "0" } + - { label: "3.11 xdist ctrace", python-version: "3.11", workers: "auto" } + env: + COVERAGE_CORE: ctrace + steps: + - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + with: + fetch-depth: 0 + - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: ${{ matrix.python-version }} + - run: sudo apt-get update && sudo apt-get install -y graphviz + - run: uv sync --locked --all-extras --dev --python ${{ matrix.python-version }} + - name: Run tests + run: uv run --python ${{ matrix.python-version }} pytest -n ${{ matrix.workers }} -q -rsxX --durations=20 -m "not postgres" --cov=src --cov-report=term-missing license-check: uses: ./.github/workflows/_license-check.yml diff --git a/.zed/rules b/.zed/rules index ac2e76dd2..42ae0172d 100644 --- a/.zed/rules +++ b/.zed/rules @@ -247,18 +247,37 @@ Developer decisions, November 12, 2026 (ITL-1007); the full procedure is RELEASI Release workflow, or touches a setting, secret or environment there. The push happens only on the developer's explicit go. The v0.2.0 tag is never pushed. - Linear, Codecov and SpiralDB jobs run only when - github.repository == 'metamorphic-labs/orcapod-python'; lint.yml runs only on nauticalab. + github.repository == 'metamorphic-labs/orcapod-python'; lint.yml runs on both repositories (ITL-1181). ## Lint, format and type checks -Every pull request must pass the metamorphic-labs required CI, which runs exactly these -commands (pytest on Python 3.11, 3.12 and 3.13; the others on 3.11): +Every pull request must pass these checks; CI runs exactly these commands: uv run --with ruff ruff check . uv run --with ruff ruff format --check . uv run --with basedpyright basedpyright uv run pytest tests/ +What runs where (developer, November 16, 2026; ITL-1181): + +- The metamorphic-labs required CI (organization ruleset `python-required-ci`, a shared + workflow this repository cannot change) runs all four on every pull request: pytest on + Python 3.11, 3.12 and 3.13, the others on 3.11. It applies only while the repository is + private (ITL-1193). +- `lint.yml` (job `Ruff and basedpyright`) runs the first three on every pull request and + push to `main`, on both repositories. `run-tests.yml` runs pytest with coverage: on a pull + request one job, `test (3.13)`, whose coverage is the one reported to Codecov; on a push + to `main`, Python 3.11, 3.12 and 3.13. Once the required CI no longer applies, a pull + request is tested on Python 3.13 only, and a failure only 3.11 or 3.12 shows is first + seen on `main`. +- In CI the tests run in parallel on every core (pytest-xdist, `-n auto`). One place turns + it on, `tests/conftest.py`, when `CI` is `true` and no `-n` is given; `-n 0` keeps a run + serial (the PostgreSQL and SpiralDB integration jobs). Coverage on Python 3.12 and later + uses `COVERAGE_CORE=sysmon`. +- Locally the tests run serially. `uv run pytest -n 4 tests/` runs them on four workers; + each worker holds its own copy of the suite's memory, so do it only on a machine with + cores and memory to spare, never beside another heavy process. + - New and changed code must be ruff-clean and ruff-formatted, and must add no basedpyright errors. Warnings are reported but do not fail the check (`failOnWarnings = false`); still do not add them needlessly. Run diff --git a/CLAUDE.md b/CLAUDE.md index 3c94b710b..f9b10cbfc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -256,12 +256,11 @@ Developer decisions, November 12, 2026 (ITL-1007); the full procedure is `RELEAS Release workflow, or touches a setting, secret or environment there. The push happens only on the developer's explicit go. The `v0.2.0` tag is never pushed. - Linear, Codecov and SpiralDB jobs run only when - `github.repository == 'metamorphic-labs/orcapod-python'`; `lint.yml` runs only on `nauticalab`. + `github.repository == 'metamorphic-labs/orcapod-python'`; `lint.yml` runs on both repositories (ITL-1181). ## Lint, format and type checks -Every pull request must pass the metamorphic-labs required CI, which runs exactly these -commands (pytest on Python 3.11, 3.12 and 3.13; the others on 3.11): +Every pull request must pass these checks; CI runs exactly these commands: ``` uv run --with ruff ruff check . @@ -270,6 +269,26 @@ uv run --with basedpyright basedpyright uv run pytest tests/ ``` +What runs where (developer, November 16, 2026; ITL-1181): + +- The metamorphic-labs required CI (organization ruleset `python-required-ci`, a shared + workflow this repository cannot change) runs all four on every pull request: pytest on + Python 3.11, 3.12 and 3.13, the others on 3.11. It applies only while the repository is + private (ITL-1193). +- `lint.yml` (job `Ruff and basedpyright`) runs the first three on every pull request and + push to `main`, on both repositories. `run-tests.yml` runs pytest with coverage: on a pull + request one job, `test (3.13)`, whose coverage is the one reported to Codecov; on a push + to `main`, Python 3.11, 3.12 and 3.13. Once the required CI no longer applies, a pull + request is tested on Python 3.13 only, and a failure only 3.11 or 3.12 shows is first + seen on `main`. +- In CI the tests run in parallel on every core (pytest-xdist, `-n auto`). One place turns + it on, `tests/conftest.py`, when `CI` is `true` and no `-n` is given; `-n 0` keeps a run + serial (the PostgreSQL and SpiralDB integration jobs). Coverage on Python 3.12 and later + uses `COVERAGE_CORE=sysmon`. +- Locally the tests run serially. `uv run pytest -n 4 tests/` runs them on four workers; + each worker holds its own copy of the suite's memory, so do it only on a machine with + cores and memory to spare, never beside another heavy process. + - New and changed code must be ruff-clean and ruff-formatted, and must add no basedpyright errors. Warnings are reported but do not fail the check (`failOnWarnings = false`); still do not add them needlessly. Run diff --git a/README.md b/README.md index 475e9812c..53d06ad05 100644 --- a/README.md +++ b/README.md @@ -142,7 +142,7 @@ The project uses [uv](https://docs.astral.sh/uv/): uv sync --all-extras --dev ``` -Every pull request must pass these checks (pytest on Python 3.11, 3.12 and 3.13): +Every pull request must pass these checks: ```bash uv run --with ruff ruff check . @@ -151,6 +151,10 @@ uv run --with basedpyright basedpyright uv run pytest tests/ ``` +CI runs the tests in parallel (pytest-xdist). Locally they run serially; `uv run pytest -n 4 tests/` +runs them on four workers, each holding its own copy of the suite's memory, so use it only on a +machine with cores and memory to spare. + The PostgreSQL integration tests are skipped unless asked for (`uv run pytest -m postgres --postgres`) and need a running PostgreSQL server, reached through the standard `PG*` environment variables; see diff --git a/changelog.d/ITL-1181.md b/changelog.d/ITL-1181.md new file mode 100644 index 000000000..a50af7cf8 --- /dev/null +++ b/changelog.d/ITL-1181.md @@ -0,0 +1,17 @@ +### Changed + +- **Faster CI rounds** — CI runs the tests in parallel on every core of the runner + (pytest-xdist, a new development dependency), turned on by `tests/conftest.py` when `CI` is + `true` and no `-n` is given; local runs stay serial (`uv run pytest -n 4 tests/` runs them + in parallel). A pull request runs the coverage job (`run-tests.yml`) once, on Python 3.13 + with `COVERAGE_CORE=sysmon`; a push to `main` runs it on Python 3.11, 3.12 and 3.13. The + lint and type check workflow (`lint.yml`) now runs on the development repository too. The + Release workflow's tests run in parallel. (ITL-1181) + +### Fixed + +- **Interrupt tests in a background process** — the tests in + `tests/test_pipeline/test_job_run_in_event_loop.py` that press the notebook's interrupt + button (`_thread.interrupt_main`) install Python's default SIGINT handler for their + duration, so they pass in a process that ignores SIGINT (one started in the background). + (ITL-1181) diff --git a/pyproject.toml b/pyproject.toml index ce7b93091..fe687b3df 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -99,6 +99,7 @@ dev = [ "pytest>=8.3.5", "pytest-asyncio>=1.3.0", "pytest-cov>=6.1.1", + "pytest-xdist>=3.6", "psycopg[binary]>=3.0", "ray[default]==2.48.0", "redis>=6.2.0", diff --git a/tests/conftest.py b/tests/conftest.py index 0e3a2abb2..02c53543b 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -4,13 +4,44 @@ tests by default. Pass ``--postgres`` to opt in:: uv run pytest --postgres + +In CI (``CI`` set to ``true``, as GitHub Actions does) the tests run in parallel on every +core of the runner (pytest-xdist, ``-n auto``) unless ``-n`` is given (ITL-1181). This is +the one place that turns it on without ``-n``: it is how the organization's required pytest +check, whose command this repository cannot change, runs in parallel. Local runs stay +serial; pass ``-n 4`` to run in parallel on a machine with cores and memory to spare. """ from __future__ import annotations +import os + import pytest +def _running_in_ci() -> bool: + """Whether the tests run in CI: ``CI`` is ``true`` (or ``1``), as GitHub Actions sets it.""" + return os.environ.get("CI", "").strip().lower() in {"true", "1"} + + +@pytest.hookimpl(tryfirst=True) +def pytest_cmdline_main(config: pytest.Config) -> None: + """In CI, run the tests on every core unless ``-n`` was given (ITL-1181). + + Sets ``-n auto`` before pytest-xdist reads the option: both hooks are ``tryfirst``, + and pytest calls this conftest's before the plugin's, which was registered earlier. + ``-n 0`` (or ``-p no:xdist``) keeps a CI run serial. A worker is left alone: xdist + clears ``-n`` in each worker before calling this hook there, and setting it again + would make every worker start workers of its own. + """ + if not _running_in_ci() or not config.pluginmanager.has_plugin("xdist"): + return + if hasattr(config, "workerinput"): + return + if config.getoption("numprocesses") is None: + config.option.numprocesses = "auto" + + def pytest_addoption(parser: pytest.Parser) -> None: parser.addoption( "--postgres", diff --git a/tests/test_pipeline/test_job_run_in_event_loop.py b/tests/test_pipeline/test_job_run_in_event_loop.py index e43ae116e..d989eca52 100644 --- a/tests/test_pipeline/test_job_run_in_event_loop.py +++ b/tests/test_pipeline/test_job_run_in_event_loop.py @@ -9,10 +9,11 @@ import _thread import asyncio +import signal import threading import time import warnings -from collections.abc import Callable +from collections.abc import Callable, Iterator from pathlib import Path from typing import Any @@ -187,6 +188,22 @@ def test_run_async_refuses_the_sync_orchestrator() -> None: ) +@pytest.fixture +def interrupt_raises_keyboard_interrupt() -> Iterator[None]: + """Make SIGINT raise ``KeyboardInterrupt`` for the test, as it does in a kernel (ITL-1181). + + ``_thread.interrupt_main`` stands for the notebook's interrupt button. It does nothing + when SIGINT is ignored, as it is in a process started in the background (a shell's + ``&``, some CI runners and test workers), so the tests that press the button install + Python's default handler for their duration and restore the one they found. + """ + previous = signal.signal(signal.SIGINT, signal.default_int_handler) + try: + yield + finally: + _ = signal.signal(signal.SIGINT, previous) + + N_SLOW_ROWS = 200 # more than the default thread pool runs at once slow_calls: list[int] = [] @@ -201,6 +218,7 @@ def slow_and_interrupted(x: int) -> int: return x +@pytest.mark.usefixtures("interrupt_raises_keyboard_interrupt") def test_interrupt_while_running_inside_a_loop_stops_the_run() -> None: """The notebook's interrupt button stops the run; nothing keeps running after it.""" slow_calls.clear() @@ -306,6 +324,7 @@ def bridge_threads() -> list[threading.Thread]: ] +@pytest.mark.usefixtures("interrupt_raises_keyboard_interrupt") @pytest.mark.parametrize("store_kind", list(STORES)) def test_second_interrupt_returns_and_a_late_function_writes_nothing( store_kind: str, tmp_path: Path @@ -384,6 +403,7 @@ async def async_stuck_on_first_row(x: int) -> int: } +@pytest.mark.usefixtures("interrupt_raises_keyboard_interrupt") @pytest.mark.parametrize("pod_form", list(POD_FORMS)) def test_second_interrupt_names_an_async_function_blocking_the_loop( pod_form: str, diff --git a/uv.lock b/uv.lock index fa31d7802..8438156af 100644 --- a/uv.lock +++ b/uv.lock @@ -850,6 +850,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/de/15/545e2b6cf2e3be84bc1ed85613edd75b8aea69807a71c26f4ca6a9258e82/email_validator-2.3.0-py3-none-any.whl", hash = "sha256:80f13f623413e6b197ae73bb10bf4eb0908faf509ad8362c5edeb0be7fd450b4", size = 35604, upload-time = "2025-08-26T13:09:05.858Z" }, ] +[[package]] +name = "execnet" +version = "2.1.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/bf/89/780e11f9588d9e7128a3f87788354c7946a9cbb1401ad38a48c4db9a4f07/execnet-2.1.2.tar.gz", hash = "sha256:63d83bfdd9a23e35b9c6a3261412324f964c2ec8dcd8d3c6916ee9373e0befcd", size = 166622, upload-time = "2025-11-12T09:56:37.75Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ab/84/02fc1827e8cdded4aa65baef11296a9bbe595c474f0d6d758af082d849fd/execnet-2.1.2-py3-none-any.whl", hash = "sha256:67fba928dd5a544b783f6056f449e5e3931a5c378b128bc18501f7ea79e296ec", size = 40708, upload-time = "2025-11-12T09:56:36.333Z" }, +] + [[package]] name = "executing" version = "2.2.0" @@ -2367,6 +2376,7 @@ dev = [ { name = "pytest" }, { name = "pytest-asyncio" }, { name = "pytest-cov" }, + { name = "pytest-xdist" }, { name = "ray", extra = ["default"] }, { name = "redis" }, { name = "ruff" }, @@ -2438,6 +2448,7 @@ dev = [ { name = "pytest", specifier = ">=8.3.5" }, { name = "pytest-asyncio", specifier = ">=1.3.0" }, { name = "pytest-cov", specifier = ">=6.1.1" }, + { name = "pytest-xdist", specifier = ">=3.6" }, { name = "ray", extras = ["default"], specifier = "==2.48.0" }, { name = "redis", specifier = ">=6.2.0" }, { name = "ruff", specifier = ">=0.14.4" }, @@ -3412,6 +3423,19 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/28/d0/def53b4a790cfb21483016430ed828f64830dd981ebe1089971cd10cab25/pytest_cov-6.1.1-py3-none-any.whl", hash = "sha256:bddf29ed2d0ab6f4df17b4c55b0a657287db8684af9c42ea546b21b1041b3dde", size = 23841, upload-time = "2025-04-05T14:07:49.641Z" }, ] +[[package]] +name = "pytest-xdist" +version = "3.8.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "execnet" }, + { name = "pytest" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/78/b4/439b179d1ff526791eb921115fca8e44e596a13efeda518b9d845a619450/pytest_xdist-3.8.0.tar.gz", hash = "sha256:7e578125ec9bc6050861aa93f2d59f1d8d085595d6551c2c90b6f4fad8d3a9f1", size = 88069, upload-time = "2025-07-01T13:30:59.346Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ca/31/d4e37e9e550c2b92a9cbc2e4d0b7420a27224968580b5a447f420847c975/pytest_xdist-3.8.0-py3-none-any.whl", hash = "sha256:202ca578cfeb7370784a8c33d6d05bc6e13b4f25b5053c30a152269fd10f0b88", size = 46396, upload-time = "2025-07-01T13:30:56.632Z" }, +] + [[package]] name = "python-dateutil" version = "2.9.0.post0" From 4e285a9b4aab9ed0d8febb89da8bb26f4aca31d0 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Mon, 16 Nov 2026 23:04:02 -0800 Subject: [PATCH 2/3] ci(tests): one xdist worker per logical CPU, not per physical core (ITL-1181) -n auto counts physical cores: a GitHub runner has one, so every CI job of the first round ran on one worker. Adds the docs/dev spec and the per-file timing script. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/release.yml | 2 +- .github/workflows/run-tests.yml | 4 +- .zed/rules | 9 +- CLAUDE.md | 9 +- docs/dev/scripts/itl_1181_per_file_times.py | 64 +++++++++++++ .../2026-11-16-itl-1181-faster-ci-rounds.md | 93 +++++++++++++++++++ tests/conftest.py | 11 ++- 7 files changed, 176 insertions(+), 16 deletions(-) create mode 100644 docs/dev/scripts/itl_1181_per_file_times.py create mode 100644 docs/dev/specs/2026-11-16-itl-1181-faster-ci-rounds.md diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 7111d89f3..1ecf2598d 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -66,7 +66,7 @@ jobs: run: uv sync --locked --all-extras --dev --python ${{ matrix.python-version }} - name: Run tests - run: uv run --python ${{ matrix.python-version }} pytest -n auto -m "not postgres" --tb=short -q + run: uv run --python ${{ matrix.python-version }} pytest -n logical -m "not postgres" --tb=short -q license-check: needs: [validate-branch] diff --git a/.github/workflows/run-tests.yml b/.github/workflows/run-tests.yml index 1dda9fad8..c06c9bc98 100644 --- a/.github/workflows/run-tests.yml +++ b/.github/workflows/run-tests.yml @@ -48,7 +48,7 @@ jobs: - name: Run tests # In parallel on every core of the runner (pytest-xdist; ITL-1181); pytest-cov # combines the workers' coverage into one report. - run: uv run --python ${{ matrix.python-version }} pytest -n auto -v -rsxX --durations=20 -m "not postgres" --cov=src --cov-report=term-missing --cov-report=xml + run: uv run --python ${{ matrix.python-version }} pytest -n logical -v -rsxX --durations=20 -m "not postgres" --cov=src --cov-report=term-missing --cov-report=xml - name: Upload coverage reports to Codecov if: github.repository == 'metamorphic-labs/orcapod-python' @@ -154,7 +154,7 @@ jobs: matrix: include: - { label: "3.13 serial ctrace", python-version: "3.13", workers: "0" } - - { label: "3.11 xdist ctrace", python-version: "3.11", workers: "auto" } + - { label: "3.11 xdist ctrace", python-version: "3.11", workers: "logical" } env: COVERAGE_CORE: ctrace steps: diff --git a/.zed/rules b/.zed/rules index 42ae0172d..470a2c194 100644 --- a/.zed/rules +++ b/.zed/rules @@ -270,10 +270,11 @@ What runs where (developer, November 16, 2026; ITL-1181): to `main`, Python 3.11, 3.12 and 3.13. Once the required CI no longer applies, a pull request is tested on Python 3.13 only, and a failure only 3.11 or 3.12 shows is first seen on `main`. -- In CI the tests run in parallel on every core (pytest-xdist, `-n auto`). One place turns - it on, `tests/conftest.py`, when `CI` is `true` and no `-n` is given; `-n 0` keeps a run - serial (the PostgreSQL and SpiralDB integration jobs). Coverage on Python 3.12 and later - uses `COVERAGE_CORE=sysmon`. +- In CI the tests run in parallel, one worker per logical CPU (pytest-xdist, `-n logical`; + `-n auto` counts physical cores, one on a GitHub runner). One place turns it on, + `tests/conftest.py`, when `CI` is `true` and no `-n` is given; `-n 0` keeps a run serial + (the PostgreSQL and SpiralDB integration jobs). Coverage on Python 3.12 and later uses + `COVERAGE_CORE=sysmon`. - Locally the tests run serially. `uv run pytest -n 4 tests/` runs them on four workers; each worker holds its own copy of the suite's memory, so do it only on a machine with cores and memory to spare, never beside another heavy process. diff --git a/CLAUDE.md b/CLAUDE.md index f9b10cbfc..63f2e0c2d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -281,10 +281,11 @@ What runs where (developer, November 16, 2026; ITL-1181): to `main`, Python 3.11, 3.12 and 3.13. Once the required CI no longer applies, a pull request is tested on Python 3.13 only, and a failure only 3.11 or 3.12 shows is first seen on `main`. -- In CI the tests run in parallel on every core (pytest-xdist, `-n auto`). One place turns - it on, `tests/conftest.py`, when `CI` is `true` and no `-n` is given; `-n 0` keeps a run - serial (the PostgreSQL and SpiralDB integration jobs). Coverage on Python 3.12 and later - uses `COVERAGE_CORE=sysmon`. +- In CI the tests run in parallel, one worker per logical CPU (pytest-xdist, `-n logical`; + `-n auto` counts physical cores, one on a GitHub runner). One place turns it on, + `tests/conftest.py`, when `CI` is `true` and no `-n` is given; `-n 0` keeps a run serial + (the PostgreSQL and SpiralDB integration jobs). Coverage on Python 3.12 and later uses + `COVERAGE_CORE=sysmon`. - Locally the tests run serially. `uv run pytest -n 4 tests/` runs them on four workers; each worker holds its own copy of the suite's memory, so do it only on a machine with cores and memory to spare, never beside another heavy process. diff --git a/docs/dev/scripts/itl_1181_per_file_times.py b/docs/dev/scripts/itl_1181_per_file_times.py new file mode 100644 index 000000000..f74d62cbb --- /dev/null +++ b/docs/dev/scripts/itl_1181_per_file_times.py @@ -0,0 +1,64 @@ +"""Sum test time per test file from pytest JUnit XML reports (ITL-1181). + +The organization's required pytest check uploads one report per Python version +(``test-results-py3.X/test-results.xml``); download them with +``gh run download `` and pass their paths. Prints the files by total time over +all reports, with each file's share and the running share, leaving out the files that +protect what CI must always check (golden hashes, the hash label rule, the documentation +samples and build, the changelog and design-issue checks, the hash site inventory). + + uv run python docs/dev/scripts/itl_1181_per_file_times.py test-results-py3.*/test-results.xml +""" + +from __future__ import annotations + +import collections +import sys +import xml.etree.ElementTree as ET + +PROTECTED = ( + "tests/test_hashing/golden/", + "test_hash_label_rule.py", + "test_hash_site_inventory.py", + "test_doc_samples.py", + "test_mkdocs_build.py", + "test_changelog_fragments.py", + "test_design_issues.py", +) + + +def module_path(classname: str) -> str: + """The test file of a JUnit ``classname`` (``tests.a.test_b.TestC`` to ``tests/a/test_b.py``).""" + parts = classname.split(".") + while parts and parts[-1][:1].isupper(): + _ = parts.pop() + return "/".join(parts) + ".py" + + +def main(paths: list[str], top: int = 19) -> None: + """Print the ``top`` slowest unprotected files over the reports at ``paths``.""" + seconds: collections.Counter[str] = collections.Counter() + total = 0.0 + for path in paths: + for case in ET.parse(path).getroot().iter("testcase"): + duration = float(case.get("time") or 0) + seconds[module_path(case.get("classname", ""))] += duration + total += duration + print(f"{len(paths)} report(s), {total / len(paths):.0f} s of test time per report") + running = 0.0 + shown = 0 + for name, spent in seconds.most_common(): + if any(marker in name for marker in PROTECTED): + continue + running += spent + shown += 1 + print( + f"{shown:2d} {spent / len(paths):6.1f} s {100 * spent / total:4.1f} %" + + f" (running {100 * running / total:5.1f} %) {name}" + ) + if shown == top: + break + + +if __name__ == "__main__": + main(sys.argv[1:]) diff --git a/docs/dev/specs/2026-11-16-itl-1181-faster-ci-rounds.md b/docs/dev/specs/2026-11-16-itl-1181-faster-ci-rounds.md new file mode 100644 index 000000000..99c3852f5 --- /dev/null +++ b/docs/dev/specs/2026-11-16-itl-1181-faster-ci-rounds.md @@ -0,0 +1,93 @@ +# Faster CI rounds: parallel tests, one coverage job per pull request (ITL-1181) + +Developer decision, November 16, 2026: "run CI in parallel with only one coverage to be +reported and full three versions only on pushes to main". The test suite is not split; the +split ITL-1181 first asked for is kept below as a reserve plan. + +## The problem, measured + +Before this change every pull request ran the whole suite (about 25,000 tests) twice on each +of Python 3.11, 3.12 and 3.13, serially: + +- the organization's required check, `pytest / pytest (3.x)` (shared workflow + `metamorphic-labs/shared-workflows`, `ci-pytest.yml`, running `uv run pytest tests`): 10 to + 19 minutes per job. This repository cannot change its command; only `pytest.ini` and + `conftest.py` reach it. +- this repository's `run-tests.yml`, `test (3.x)`, with coverage: 19 to 37 minutes per job. + Coverage cost +55 % on Python 3.11 and about 2.5 times on 3.12 and 3.13 with the default + tracer (`COVERAGE_CORE=sysmon` removes most of it on 3.12 and later). + +A round of review on PR #416 (the last pull request before this one) took 30 minutes from +push to the last check: the `test (3.12)` coverage job, 29 minutes. + +Locally, serial: 963 s. With pytest-xdist on four workers: 261 s, 24,842 passed, no failure +that only the parallel run shows (measured by the previous worker on ITL-1181). + +## What changed + +1. pytest-xdist is a development dependency. `tests/conftest.py` (`pytest_cmdline_main`) + sets `-n logical` when `CI` is `true` and no `-n` was given, and never on an xdist worker + (xdist clears `-n` in a worker before calling the hook there; setting it again would + make every worker start workers). That is the one place it is turned on, and the way the + organization's required check runs in parallel. Local runs stay serial. +2. `run-tests.yml`: a pull request runs one job, `test (3.13)`, with + `COVERAGE_CORE=sysmon` and `-n logical`, whose coverage is the one reported to Codecov. A + push (to `main`, `dev`, `extension-type-system`) and a manual run use Python 3.11, 3.12 + and 3.13, with coverage as before (3.11 with the C tracer, 3.12 and 3.13 with `sysmon`). + The PostgreSQL job and the SpiralDB integration job stay serial (`-n 0`): their tests + share one server or project. +3. The tests in `tests/test_pipeline/test_job_run_in_event_loop.py` that press the + notebook's interrupt button (`_thread.interrupt_main`) install Python's default SIGINT + handler for their duration (fixture `interrupt_raises_keyboard_interrupt`). In a process + that ignores SIGINT (started in the background) `interrupt_main` does nothing, and the six + tests failed. +4. `lint.yml` (job `Ruff and basedpyright`) runs on this repository too. Once the repository + is public (ITL-1193) the organization ruleset no longer applies to it, and the + repository's own checks are all that remain. +5. `release.yml`: the test step runs with `-n logical`; nothing else changed. + +## Results on this pull request + +TBD + +## Check names + +TBD + +## Reserve plan: a full-only list (not implemented) + +If a pull-request round is still too slow, the slowest test files can run only on pushes to +`main` and in the Release workflow, kept out of the pull-request job by one list. The 19 +slowest files that protect nothing CI must always check hold 55 % of the test time (JUnit +reports of PR #416's required check, all three versions, +`docs/dev/scripts/itl_1181_per_file_times.py`): + +| # | File | Seconds per run | Share | +|---|---|---|---| +| 1 | `tests/test_core/alignment/test_nodes.py` | 85.2 | 10.7 % | +| 2 | `tests/test_pipeline/test_async_fan_out_does_not_hang.py` | 54.9 | 6.9 % | +| 3 | `tests/test_user_types/test_workshop_scenarios.py` | 39.9 | 5.0 % | +| 4 | `tests/test_pipeline/test_zero_row_column_order.py` | 38.3 | 4.8 % | +| 5 | `tests/test_core/alignment/test_operators.py` | 24.3 | 3.1 % | +| 6 | `tests/test_pipeline/test_loaded_slot_unregistered_class.py` | 22.3 | 2.8 % | +| 7 | `tests/test_databases/test_delta_exit.py` | 21.1 | 2.6 % | +| 8 | `tests/test_torch/test_tensor_conversion_warning.py` | 17.4 | 2.2 % | +| 9 | `tests/test_logical_types/test_set_of_primitives.py` | 16.3 | 2.0 % | +| 10 | `tests/test_core/alignment/test_streams.py` | 15.9 | 2.0 % | +| 11 | `tests/test_core/operators/test_large_normalization.py` | 15.4 | 1.9 % | +| 12 | `tests/test_pipeline/test_job_store_scaling.py` | 14.9 | 1.9 % | +| 13 | `tests/test_pipeline/test_job_crash_recovery.py` | 13.2 | 1.7 % | +| 14 | `tests/test_hashing/test_pandas_storage_matrix.py` | 11.8 | 1.5 % | +| 15 | `tests/test_torch/test_tensor_cached_rerun.py` | 10.6 | 1.3 % | +| 16 | `tests/test_torch/test_tensor_hash_stability.py` | 10.2 | 1.3 % | +| 17 | `tests/test_pipeline/test_job_batched_commits.py` | 10.1 | 1.3 % | +| 18 | `tests/test_core/function_pod/test_output_conformance.py` | 10.0 | 1.3 % | +| 19 | `tests/test_pipeline/test_node_head_sorting.py` | 9.6 | 1.2 % | + +The protected files are never on the list: the golden hash tests, the hash label rule, the +hash site inventory, the documentation samples and build, and the changelog and +design-issue checks. Implementing it means one list (a marker applied by `conftest.py` from +the list, deselected with `-m "not full_only"` in the pull-request job), and it cannot reach +the organization's required check, which runs `uv run pytest tests` unchanged. A file +on the list would be checked on a pull request only by that required check, while it +applies. diff --git a/tests/conftest.py b/tests/conftest.py index 02c53543b..de039dd5d 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -5,8 +5,9 @@ uv run pytest --postgres -In CI (``CI`` set to ``true``, as GitHub Actions does) the tests run in parallel on every -core of the runner (pytest-xdist, ``-n auto``) unless ``-n`` is given (ITL-1181). This is +In CI (``CI`` set to ``true``, as GitHub Actions does) the tests run in parallel, one worker +per logical CPU of the runner (pytest-xdist, ``-n logical``; ``-n auto`` counts physical +cores, of which a GitHub runner has one), unless ``-n`` is given (ITL-1181). This is the one place that turns it on without ``-n``: it is how the organization's required pytest check, whose command this repository cannot change, runs in parallel. Local runs stay serial; pass ``-n 4`` to run in parallel on a machine with cores and memory to spare. @@ -26,9 +27,9 @@ def _running_in_ci() -> bool: @pytest.hookimpl(tryfirst=True) def pytest_cmdline_main(config: pytest.Config) -> None: - """In CI, run the tests on every core unless ``-n`` was given (ITL-1181). + """In CI, run the tests on every logical CPU unless ``-n`` was given (ITL-1181). - Sets ``-n auto`` before pytest-xdist reads the option: both hooks are ``tryfirst``, + Sets ``-n logical`` before pytest-xdist reads the option: both hooks are ``tryfirst``, and pytest calls this conftest's before the plugin's, which was registered earlier. ``-n 0`` (or ``-p no:xdist``) keeps a CI run serial. A worker is left alone: xdist clears ``-n`` in each worker before calling this hook there, and setting it again @@ -39,7 +40,7 @@ def pytest_cmdline_main(config: pytest.Config) -> None: if hasattr(config, "workerinput"): return if config.getoption("numprocesses") is None: - config.option.numprocesses = "auto" + config.option.numprocesses = "logical" def pytest_addoption(parser: pytest.Parser) -> None: From 8fa404b75d5861f7fafca593e445b43c2e651db1 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Mon, 16 Nov 2026 23:39:03 -0800 Subject: [PATCH 3/3] ci(tests): remove the temporary measurement jobs; record the results (ITL-1181) Co-Authored-By: Claude Opus 5.5 --- .github/workflows/run-tests.yml | 30 ----------- docs/dev/scripts/itl_1181_per_file_times.py | 4 +- .../2026-11-16-itl-1181-faster-ci-rounds.md | 52 ++++++++++++++++++- 3 files changed, 52 insertions(+), 34 deletions(-) diff --git a/.github/workflows/run-tests.yml b/.github/workflows/run-tests.yml index c06c9bc98..d715b12ad 100644 --- a/.github/workflows/run-tests.yml +++ b/.github/workflows/run-tests.yml @@ -140,36 +140,6 @@ jobs: # Serial (-n 0): the integration tests share one SpiralDB project (ITL-1181). run: uv run --python "3.11" pytest -n 0 tests/test_databases/test_spiraldb_connector_integration.py -v - # TEMPORARY (ITL-1181): measurements for the pull request; removed before it merges. - # serial-ctrace is today's configuration (serial, default tracer), whose coverage total the - # parallel test (3.13) job's must equal; xdist-ctrace is the cost of a pull-request job on - # Python 3.11 instead of 3.13. - measure: - name: measure (${{ matrix.label }}) - if: github.event_name == 'pull_request' - needs: license-check - runs-on: ubuntu-latest - strategy: - fail-fast: false - matrix: - include: - - { label: "3.13 serial ctrace", python-version: "3.13", workers: "0" } - - { label: "3.11 xdist ctrace", python-version: "3.11", workers: "logical" } - env: - COVERAGE_CORE: ctrace - steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 - with: - fetch-depth: 0 - - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 - - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 - with: - python-version: ${{ matrix.python-version }} - - run: sudo apt-get update && sudo apt-get install -y graphviz - - run: uv sync --locked --all-extras --dev --python ${{ matrix.python-version }} - - name: Run tests - run: uv run --python ${{ matrix.python-version }} pytest -n ${{ matrix.workers }} -q -rsxX --durations=20 -m "not postgres" --cov=src --cov-report=term-missing - license-check: uses: ./.github/workflows/_license-check.yml diff --git a/docs/dev/scripts/itl_1181_per_file_times.py b/docs/dev/scripts/itl_1181_per_file_times.py index f74d62cbb..f7ce9db1e 100644 --- a/docs/dev/scripts/itl_1181_per_file_times.py +++ b/docs/dev/scripts/itl_1181_per_file_times.py @@ -37,7 +37,7 @@ def module_path(classname: str) -> str: def main(paths: list[str], top: int = 19) -> None: """Print the ``top`` slowest unprotected files over the reports at ``paths``.""" - seconds: collections.Counter[str] = collections.Counter() + seconds: collections.defaultdict[str, float] = collections.defaultdict(float) total = 0.0 for path in paths: for case in ET.parse(path).getroot().iter("testcase"): @@ -47,7 +47,7 @@ def main(paths: list[str], top: int = 19) -> None: print(f"{len(paths)} report(s), {total / len(paths):.0f} s of test time per report") running = 0.0 shown = 0 - for name, spent in seconds.most_common(): + for name, spent in sorted(seconds.items(), key=lambda item: -item[1]): if any(marker in name for marker in PROTECTED): continue running += spent diff --git a/docs/dev/specs/2026-11-16-itl-1181-faster-ci-rounds.md b/docs/dev/specs/2026-11-16-itl-1181-faster-ci-rounds.md index 99c3852f5..3b8408d68 100644 --- a/docs/dev/specs/2026-11-16-itl-1181-faster-ci-rounds.md +++ b/docs/dev/specs/2026-11-16-itl-1181-faster-ci-rounds.md @@ -48,11 +48,59 @@ that only the parallel run shows (measured by the previous worker on ITL-1181). ## Results on this pull request -TBD +A GitHub runner for a private repository has two logical CPUs and one physical core, so +`-n auto` (physical cores) started one worker on the first round; `-n logical` starts two. +Test time is pytest's own figure; a job adds two to three minutes of setup. Runner speed +varies between rounds by up to a third (the same serial job took 1580 s and 1195 s). + +| Job | Before (PR #416, serial) | Round 1 (1 worker) | Round 2 (2 workers) | +|---|---|---|---| +| `pytest / pytest (3.11)` (required) | 17.7 min job | 852 s | 697 s | +| `pytest / pytest (3.12)` (required) | 12.2 min job | 1071 s | 669 s | +| `pytest / pytest (3.13)` (required) | 13.8 min job | 799 s | 835 s | +| `test (3.13)`, coverage | 20.7 min job (C tracer) | 737 s (`sysmon`) | 848 s (`sysmon`) | +| `test (3.11)` / `test (3.12)`, coverage | 25.2 / 28.8 min jobs | not run on a pull request | not run on a pull request | + +A round (push to the last test check): 30 minutes on PR #416, 16 minutes on round 2 of this +pull request. Most of the gain is the coverage jobs: one job instead of three, and +`sysmon` halves the 3.13 job (serial, C tracer: 1580 s and 1195 s; parallel, `sysmon`: +737 s and 848 s). Two workers on one physical core gain 15 to 35 % on the required check. +A public repository's runners have four logical CPUs, so the gain grows there. + +Coverage, measured once on this pull request against a temporary serial job on Python 3.13 +with the C tracer (today's configuration): 3,306 of 27,404 statements missed serially and +3,307 in the parallel `sysmon` job, both 88 %. The one line is +`src/orcapod/hashing/cache_population.py:393`, a timing-dependent branch; no file differs +otherwise. + +The pull-request job on Python 3.11 with the C tracer instead of 3.13 with `sysmon` +(temporary job): 1256 s (round 1, one worker) and 1849 s (round 2, two workers) of test time, against +737 s and 848 s; about twice as long, 21 to 31 minutes per round instead of 12 to 14. + +ITL-1192 (`test_second_interrupt_returns_and_a_late_function_writes_nothing[delta]`, the +warning naming nodes `stuck`, `after`) failed once in round 2, in the temporary 3.11 job +(two workers with the C tracer, the most loaded job); it failed before this change too, +serially, on 3.13. More load per CPU makes the race more likely; the fix is ITL-1192. ## Check names -TBD +A pull request reports, while the organization ruleset applies (private repository): + +- Python — Required CI: `resolve-matrix / resolve`, `pytest / pytest (3.11)`, + `pytest / pytest (3.12)`, `pytest / pytest (3.13)`, `ruff / ruff`, + `typecheck / typecheck`, `license-check / license-check` (the required ones). +- Run Tests: `license-check / License check`, `Dependency review`, `test (3.13)`, + `spiral-integration` (skipped on a pull request). +- Lint and type check: `Ruff and basedpyright` (new here; skipped before this change). +- Run Objective Tests: `test (3.11)`, `test (3.12)`, `test (3.13)`. +- Run PostgreSQL Tests: `test (3.11)`, `test (3.12)`, `test (3.13)`. +- `codecov/patch`, `codecov/project`, `CodeRabbit`. + +Before this change the pull request also reported Run Tests `test (3.11)` and +`test (3.12)`. After the repository is public (ITL-1193) the Python — Required CI checks are +gone and the rest remain; the developer then chooses which to require. Tests then run on a +pull request on Python 3.13 only (`test (3.13)`), and a failure only 3.11 or 3.12 shows is +first seen on `main`. ## Reserve plan: a full-only list (not implemented)