Skip to content

ci(conda-lock): install py-rattler and pyyaml from a hashed tool lock - #771

Open
antonwolfy wants to merge 2 commits into
intel:mainfrom
antonwolfy:ci/conda-lock-tool-lock
Open

antonwolfy wants to merge 2 commits into
intel:mainfrom
antonwolfy:ci/conda-lock-tool-lock

Conversation

@antonwolfy

Copy link
Copy Markdown
Contributor

Summary

#681: .github/workflows/conda-lock.yml runs python -m pip install --no-input "py-rattler ==0.28.1" "pyyaml ==6.0.3" in the one job that holds contents: write, pull-requests: write and CONDA_LOCK_TOKEN, and the code it installs writes the lock the refresh PR proposes. On main the two versions are pinned but their file hashes are not, and py-rattler's and PyYAML's own dependencies are not pinned at all, while ci.yml and pip-audit.yml install their tools from a hashed lock with --require-hashes.

The fix is the one the issue names: a requirements input for the two tools (dokimi/requirements-conda-lock.txt), a hashed uv pip compile --universal --generate-hashes --exclude-newer=P7D lock next to the others (dokimi/requirements-conda-lock.lock.txt, compiled for the job's Python 3.14), and an install with pip install --no-deps --require-hashes -r dokimi/requirements-conda-lock.lock.txt. Renovate's pip-compile manager takes the lock in place of the regex manager that read the pins out of conda-lock.yml. The lock closes over exactly the two pins (py-rattler has no Python dependencies to pin; PyYAML none), so --no-deps loses nothing.

Closes #681

Changes

  • dokimi/requirements-conda-lock.txt (py-rattler 0.28.1, pyyaml 6.0.3, the versions main installs) and dokimi/requirements-conda-lock.lock.txt, whose header is the uv pip compile command Renovate re-runs.
  • .github/workflows/conda-lock.yml: the install step reads the lock with --no-deps --require-hashes, by the repo-relative path the job's working directory needs; the refresh job's other steps are untouched.
  • .github/renovate.json5: the pip-compile pattern covers requirements-conda-lock.lock.txt; the regex custom manager for conda-lock.yml is deleted; the header comment lists the new lock.
  • .github/workflows/pip-audit.yml audits the lock, so an advisory on a pin of the refresh job is seen; .github/.licenserc.yaml exempts it from the header as the sibling locks are.
  • ci/lock.py (docstring) and ci/README.md (the lock.py row and the re-solve instruction) name the lock instead of an unhashed pip install.
  • dokimi/tests/test_dependency_locks.py: the lock joins the Renovate, hashed-lock, header-form, cooldown, hash and pip-audit checks of its siblings; new tests pin that its pins equal the input's, that conda-lock.yml installs it with --no-deps --require-hashes by an existing repo-relative path, that no workflow runs pip install of py-rattler without --require-hashes, and that Renovate's pip-compile pattern matches the lock while no regex manager names conda-lock.yml.
  • No changelog fragment: the static job's Fragment for package changes step prints "no change to the package, dokimi/data/ or what pip installs" for this diff.

Risk

The refresh job installs the same two versions as before, now with hashes and without pip resolving anything. If a wheel for the runner is missing from the lock (the lock is universal, so every platform's wheel hashes are in it) the install fails instead of falling back to an unhashed download. Renovate now opens a pip-compile PR for the lock rather than editing the workflow text, and the next bump changes the lock and the input together; the pins in the input and the lock must stay equal, which a test checks. The Renovate behaviour itself (the new manager run) is not exercised before merge.

Testing

On Linux, at e550b77 (on main 7137604):

  • make ci-local: 33 passed, 0 failed, 14 skipped (docker-only and matrix legs, micromamba, semgrep, jobs conditional on a push or schedule). The pytest step: 20450 passed, 19 skipped.
  • On main's code with this branch's test_dependency_locks.py: 9 tests fail (the missing lock and input, the unhashed install line, the missing Renovate pattern and pip-audit entry).
  • Mutants, each failing the assertion it pins: --require-hashes dropped from the install step fails test_conda_lock_job_installs_its_tools_from_the_hashed_lock; the old unhashed pip install "py-rattler ==..." line fails that test and test_no_workflow_installs_py_rattler_without_hashes; a lock pin changed from the input's fails test_conda_lock_tool_lock_is_the_compile_of_its_input; -conda-lock dropped from the Renovate pattern, and a regex manager for conda-lock.yml put back, each fail test_renovate_recompiles_the_conda_lock_tool_lock; the lock dropped from the pip-audit command fails test_pip_audit_audits_every_lock[requirements-conda-lock.lock.txt]; --generate-hashes dropped from the header fails test_lock_is_compiled_with_hashes[requirements-conda-lock.lock.txt].
  • Not merged together with a second PR: the change for the Renovate-branch re-solve (ci(conda-lock): re-solve the conda locks on a Renovate branch that bumps a ci/config.yml pin #682) touches the same lines of .github/renovate.json5 and ci/README.md, so it is rebuilt on main once this merges.
  • Outside the build: the lock was compiled with the command in its header and is byte-identical to the committed file; pip install --no-deps --require-hashes -r of it into a fresh Python 3.14 environment succeeds and import rattler, yaml works.
  • Not run: Renovate itself, and the other Python versions.

Revert

A plain revert restores the unhashed pip install "py-rattler ==0.28.1" "pyyaml ==6.0.3" line and the regex manager, and deletes the two requirements files. Nothing is stored.

Checklist

  • Tests added/updated and pytest passes
  • ruff, mypy, and schema checks pass
  • Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
  • Self-reviewed per "Before opening a PR" in AGENTS.md
  • No ad-hoc run artifacts committed
  • New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

…intel#681)

The conda-lock refresh job holds contents: write, pull-requests: write and CONDA_LOCK_TOKEN, and the code it installs writes the lock the PR proposes, but it ran `pip install "py-rattler ==0.28.1" "pyyaml ==6.0.3"`: the versions were pinned, the file hashes and every transitive dependency were not.

dokimi/requirements-conda-lock.txt now names the two exact pins and dokimi/requirements-conda-lock.lock.txt is their `uv pip compile --universal --generate-hashes --exclude-newer=P7D` output, for the job's Python 3.14. conda-lock.yml installs it with `pip install --no-deps --require-hashes -r dokimi/requirements-conda-lock.lock.txt`, a path relative to the repo root the job runs from.

Renovate's pip-compile manager now covers the lock, so it re-runs the lock's header command on a bump, and the regex manager that read the pins out of conda-lock.yml is removed. pip-audit.yml audits the lock, .licenserc.yaml exempts it from the header as the sibling locks are, and ci/lock.py and ci/README.md name the lock instead of an unhashed install.

tests/test_dependency_locks.py puts the lock through the checks the sibling locks have (header form, hashes, cooldown, pip-audit) and adds tests that its pins are the input's, that conda-lock.yml installs it with --require-hashes, that no workflow installs py-rattler without hashes, and that Renovate's pip-compile pattern matches it while no regex manager names conda-lock.yml.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@antonwolfy
antonwolfy requested a review from a team as a code owner November 14, 2026 20:36
@antonwolfy antonwolfy self-assigned this Nov 14, 2026
@exai-bot

exai-bot commented Nov 14, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 10 selected item(s).

CI at 0b91ff1: 27 passing, 4 skipped.

Review details
  • Commit reviewed: 0b91ff1 against merge base 7137604
  • Coverage: 10 files in 4 groups
  • Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
  • Usage: 236k tokens (119k from cache), 32s
  • Tool calls: 22 (14 file_read, 2 ci_status, 2 file_find, 2 local_check_results, 1 code_search, 1 pr_comments)
  • Dependency updates and lock generation: 5 files
  • Security auditing and advisories: 3 files
  • License checking configuration: 1 file
  • Contribution guidelines: 1 file

@codecov-commenter

codecov-commenter commented Nov 14, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.72%. Comparing base (7137604) to head (0b91ff1).
⚠️ Report is 1 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #771      +/-   ##
==========================================
- Coverage   95.72%   95.72%   -0.01%     
==========================================
  Files         415      415              
  Lines       55363    55362       -1     
  Branches     9516     9516              
==========================================
- Hits        52994    52993       -1     
  Misses       1466     1466              
  Partials      903      903              

see 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…ented

SECURITY.md named five locks pip-audit covers and omitted
requirements-conda-lock.lock.txt, which pip-audit.yml audits.
CONTRIBUTING.md did not mention the lock at all. A test now fails when a
renovated lock is missing from SECURITY.md, and the advisory-list test
matches the new wording of the no-advisory sentence.

Signed-off-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com>
@vchamarthi

Copy link
Copy Markdown
Contributor

Pushed 1 commit to ci/conda-lock-tool-lock (e550b77..0b91ff1).

  • SECURITY.md lists requirements-conda-lock.lock.txt among the locks pip-audit covers, and CONTRIBUTING.md describes the conda-lock install.
    Run: the full suite with main merged in, 20496 passed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci(conda-lock): install py-rattler and pyyaml from a hashed tool lock with --require-hashes

4 participants