Skip to content

chore(ci): pipeline ci and tooling hygiene - #69

Open
kcrmin wants to merge 9 commits into
developfrom
chore/pipeline-ci-hygiene
Open

kcrmin wants to merge 9 commits into
developfrom
chore/pipeline-ci-hygiene

Conversation

@kcrmin

@kcrmin kcrmin commented Oct 27, 2026

Copy link
Copy Markdown
Member

What and why

Depends on #66 (this branch is based on build/harden-dependency-ranges). Small CI and tooling fixes for the pipeline:

  • README: the setup section named Python 3.11 and requirements.txt, and called boto3 and psycopg optional. It now describes Python 3.12 with uv or venv, and lists them as runtime dependencies.
  • gate8: sets I2S_SUPERPROJECT_ROOT to the checkout, so conftest always applies the vendored bundle. The old comment credited PIPELINE_TEST_POSTGRES_IMAGE with stopping the superproject search, but that variable does not.
  • docker-compose.localstack.yml: changed from 4.14.0 to localstack/localstack:4.7.0, the image the gate8 service runs, so gate8 and gate9 test the same emulator release.
  • New gate1b runs ruff check, ruff format and strict mypy for idea2strategy-market-loader. It installs them from the loader's uv.lock with uv sync --locked. gate1 excludes the loader, so none of these checks ran before. This also fixes the loader's one format violation in cli.py.
  • pandas-stubs moves from >=2.2,<3 to >=3,<4, matching the pandas 3 runtime.

Merge order note: #68 hash-locks the dependencies. Whichever of the two lands second must regenerate requirements/dev.txt, because the stubs pin changes.

Verified

  • gate1b steps on a clean git archive copy with uv 0.13.1: uv sync --locked, ruff check, ruff format --check, mypy all passed. ruff format --check fails on the old cli.py, and uv sync --locked fails when the loader's pyproject drifts from the lock.
  • mypy (data-pipeline) clean with pandas-stubs 3.0.0.260204 and 3.0.5.260914. ruff check . clean.
  • gate3: 1093 passed, coverage 89.10%. gate6: validator OK, 21 passed. Loader suite: 31 passed.
  • gate9 with docker compose -f docker-compose.localstack.yml up -d --wait (4.7.0, s3 and sqs running), vendored bundle: 172 executed (subtests included), 0 skipped, 0 errors.
  • gate8 with I2S_SUPERPROJECT_ROOT set to the checkout, as CI sets it, with PostgreSQL 16 Testcontainers and LocalStack 4.7.0: 160 passed, 0 skipped, 0 errors; JUnit assertion passed.

Checklist

  • gate8 and gate9 use the same LocalStack release
  • Loader lint and type checks run from its lock
  • mypy stays green on pandas-stubs 3.x
  • README setup matches requires-python

reference.sessions rejects every release but 5.4.0, and fresh installs
now resolve 5.5.0, failing gate3, gate8 and the image calendar path.
pandas-market-calendars 5.4.0 serves XNYS from exchange-calendars, so
the drift refusal now checks both releases and the pin is tested.
SQLAlchemy 2.1 retyped Result; capping the minor and pinning mypy to
the backtest-engine release keeps the type gate from moving by itself.
The setup still named 3.11 and requirements.txt, and called boto3
and psycopg optional although pyproject lists them as runtime deps.
The comment credited PIPELINE_TEST_POSTGRES_IMAGE with stopping the
superproject search; only I2S_SUPERPROJECT_ROOT does, so set it.
Compose started 4.14.0 while the gate8 service ran 4.7.0, so the two
LocalStack gates exercised different emulator releases.
ruff format --check reported one signature in cli.py that fits on
one line; it is the only file the formatter would change.
gate1 excludes the loader, so its own ruff and strict mypy settings
were never enforced. Install the tools from its uv.lock with --locked.
The runtime is pandas 3, but the stubs were capped below 3, so mypy
checked the 2.x API. mypy stays clean on 3.0.0 and 3.0.5.
@kcrmin kcrmin mentioned this pull request Oct 27, 2026
4 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant