Skip to content

build(deps): hash-lock python dependencies - #68

Open
kcrmin wants to merge 6 commits into
developfrom
build/lock-python-dependencies
Open

kcrmin wants to merge 6 commits into
developfrom
build/lock-python-dependencies

Conversation

@kcrmin

@kcrmin kcrmin commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

What and why

Depends on #66 (this branch is based on build/harden-dependency-ranges). Paired with Idea2Strategy/Idea2Strategy-backtest-engine#105, which locks backtest-engine the same way.

pyproject.toml keeps the allowed ranges; the image and CI installed whatever resolved on the day. This adds hash-locked requirement sets and installs only from them.

  • requirements/runtime.txt (41 packages) and requirements/dev.txt (runtime plus the dev extra), produced by uv pip compile --python-version 3.12 --universal --generate-hashes. The universal resolution covers the linux amd64/arm64 image and keeps platform-only packages behind markers, so macOS and Windows dev machines can use the same files. .gitignore un-ignores requirements/*.txt.
  • Dockerfile: wheels only with --require-hashes, then the project with --no-deps and pip check before the perl purge. The container contract test pins both install lines.
  • CI: every data-pipeline gate installs from the lock, then the project with --no-deps. gate5 uses the runtime lock alone. gate2 runs pip check and fails when pyproject (extras included) and requirements/ drift, or when a runtime pin differs between the two files. The market loader keeps its own uv.lock.
  • docs/IDEA2STRATEGY_DEVELOPMENT.md documents how to regenerate the lock.

Merge order note: #69 moves pandas-stubs to 3.x. Whichever of the two lands second must regenerate requirements/dev.txt. gate2's lock check reports the mismatch (pandas-stubs<4,>=3; extra == "dev": the lock pins 2.3.3.260113).

Verified

  • Image built natively on arm64 (DOCKER_BUILDKIT=0 docker build): hash-checked wheel install, pip check clean. gate5b smoke against it: arm64, imports OK, no perl-base, /ready READY, clean stop with exit 0.
  • Fresh venv from requirements/dev.txt plus --no-deps -e .: pip check clean. gate2 lock step passed, and failed as expected on a pyproject bump that was not regenerated. mypy clean, ruff clean.
  • gate3 on the dev lock: 1093 passed, coverage 89.10%.
  • gate8 on the dev lock (vendored bundle, Testcontainers PostgreSQL 16 + LocalStack 4.7.0 on 4566): 160 passed, 0 skipped, 0 errors; JUnit assertion passed. gate9 test set on the dev lock against compose LocalStack 4.7.0: 172 executed (subtests included), 0 skipped, 0 errors.
  • gate5 steps on a runtime-lock-only venv: all passed. pip-audit on that env: no known vulnerabilities.

Checklist

  • Runtime and dev pins agree for every shared package
  • Image installs third-party packages only from hashed wheels
  • CI fails on lock drift
  • Regeneration steps documented

reference.sessions rejects every release but 5.4.0, and fresh installs
now resolve 5.5.0, failing gate3, gate8 and the image calendar path.
pandas-market-calendars 5.4.0 serves XNYS from exchange-calendars, so
the drift refusal now checks both releases and the pin is tested.
SQLAlchemy 2.1 retyped Result; capping the minor and pinning mypy to
the backtest-engine release keeps the type gate from moving by itself.
runtime.txt pins what the image ships; dev.txt adds the dev extra.
The development guide documents how to regenerate both.
Wheels only and hashes required, then the project with --no-deps and
pip check, so a drifted lock fails the build instead of shipping.
gate2 now fails when pyproject.toml and requirements/ drift, extras
included, and gate5 imports the app with the runtime lock alone.
@kcrmin kcrmin mentioned this pull request Oct 27, 2026
4 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant