Repository navigation
ci(conda-lock): install py-rattler and pyyaml from a hashed tool lock - #771
Open
antonwolfy wants to merge 2 commits into
Open
antonwolfy wants to merge 2 commits into
antonwolfy wants to merge 2 commits into
Conversation
…intel#681) The conda-lock refresh job holds contents: write, pull-requests: write and CONDA_LOCK_TOKEN, and the code it installs writes the lock the PR proposes, but it ran `pip install "py-rattler ==0.28.1" "pyyaml ==6.0.3"`: the versions were pinned, the file hashes and every transitive dependency were not. dokimi/requirements-conda-lock.txt now names the two exact pins and dokimi/requirements-conda-lock.lock.txt is their `uv pip compile --universal --generate-hashes --exclude-newer=P7D` output, for the job's Python 3.14. conda-lock.yml installs it with `pip install --no-deps --require-hashes -r dokimi/requirements-conda-lock.lock.txt`, a path relative to the repo root the job runs from. Renovate's pip-compile manager now covers the lock, so it re-runs the lock's header command on a bump, and the regex manager that read the pins out of conda-lock.yml is removed. pip-audit.yml audits the lock, .licenserc.yaml exempts it from the header as the sibling locks are, and ci/lock.py and ci/README.md name the lock instead of an unhashed install. tests/test_dependency_locks.py puts the lock through the checks the sibling locks have (header form, hashes, cooldown, pip-audit) and adds tests that its pins are the input's, that conda-lock.yml installs it with --require-hashes, that no workflow installs py-rattler without hashes, and that Renovate's pip-compile pattern matches it while no regex manager names conda-lock.yml. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 10 selected item(s). CI at Review details
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #771 +/- ##
==========================================
- Coverage 95.72% 95.72% -0.01%
==========================================
Files 415 415
Lines 55363 55362 -1
Branches 9516 9516
==========================================
- Hits 52994 52993 -1
Misses 1466 1466
Partials 903 903 🚀 New features to boost your workflow:
|
…ented SECURITY.md named five locks pip-audit covers and omitted requirements-conda-lock.lock.txt, which pip-audit.yml audits. CONTRIBUTING.md did not mention the lock at all. A test now fails when a renovated lock is missing from SECURITY.md, and the advisory-list test matches the new wording of the no-advisory sentence. Signed-off-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com>
Contributor
|
Pushed 1 commit to
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
#681:
.github/workflows/conda-lock.ymlrunspython -m pip install --no-input "py-rattler ==0.28.1" "pyyaml ==6.0.3"in the one job that holdscontents: write,pull-requests: writeandCONDA_LOCK_TOKEN, and the code it installs writes the lock the refresh PR proposes. Onmainthe two versions are pinned but their file hashes are not, and py-rattler's and PyYAML's own dependencies are not pinned at all, whileci.ymlandpip-audit.ymlinstall their tools from a hashed lock with--require-hashes.The fix is the one the issue names: a requirements input for the two tools (
dokimi/requirements-conda-lock.txt), a hasheduv pip compile --universal --generate-hashes --exclude-newer=P7Dlock next to the others (dokimi/requirements-conda-lock.lock.txt, compiled for the job's Python 3.14), and an install withpip install --no-deps --require-hashes -r dokimi/requirements-conda-lock.lock.txt. Renovate'spip-compilemanager takes the lock in place of the regex manager that read the pins out ofconda-lock.yml. The lock closes over exactly the two pins (py-rattler has no Python dependencies to pin; PyYAML none), so--no-depsloses nothing.Closes #681
Changes
dokimi/requirements-conda-lock.txt(py-rattler 0.28.1, pyyaml 6.0.3, the versionsmaininstalls) anddokimi/requirements-conda-lock.lock.txt, whose header is theuv pip compilecommand Renovate re-runs..github/workflows/conda-lock.yml: the install step reads the lock with--no-deps --require-hashes, by the repo-relative path the job's working directory needs; the refresh job's other steps are untouched..github/renovate.json5: thepip-compilepattern coversrequirements-conda-lock.lock.txt; the regex custom manager forconda-lock.ymlis deleted; the header comment lists the new lock..github/workflows/pip-audit.ymlaudits the lock, so an advisory on a pin of the refresh job is seen;.github/.licenserc.yamlexempts it from the header as the sibling locks are.ci/lock.py(docstring) andci/README.md(thelock.pyrow and the re-solve instruction) name the lock instead of an unhashedpip install.dokimi/tests/test_dependency_locks.py: the lock joins the Renovate, hashed-lock, header-form, cooldown, hash and pip-audit checks of its siblings; new tests pin that its pins equal the input's, thatconda-lock.ymlinstalls it with--no-deps --require-hashesby an existing repo-relative path, that no workflow runspip installof py-rattler without--require-hashes, and that Renovate'spip-compilepattern matches the lock while no regex manager namesconda-lock.yml.staticjob'sFragment for package changesstep prints "no change to the package, dokimi/data/ or what pip installs" for this diff.Risk
The refresh job installs the same two versions as before, now with hashes and without pip resolving anything. If a wheel for the runner is missing from the lock (the lock is universal, so every platform's wheel hashes are in it) the install fails instead of falling back to an unhashed download. Renovate now opens a
pip-compilePR for the lock rather than editing the workflow text, and the next bump changes the lock and the input together; the pins in the input and the lock must stay equal, which a test checks. The Renovate behaviour itself (the new manager run) is not exercised before merge.Testing
On Linux, at
e550b77(onmain7137604):make ci-local: 33 passed, 0 failed, 14 skipped (docker-only and matrix legs,micromamba,semgrep, jobs conditional on a push or schedule). The pytest step: 20450 passed, 19 skipped.main's code with this branch'stest_dependency_locks.py: 9 tests fail (the missing lock and input, the unhashed install line, the missing Renovate pattern and pip-audit entry).--require-hashesdropped from the install step failstest_conda_lock_job_installs_its_tools_from_the_hashed_lock; the old unhashedpip install "py-rattler ==..."line fails that test andtest_no_workflow_installs_py_rattler_without_hashes; a lock pin changed from the input's failstest_conda_lock_tool_lock_is_the_compile_of_its_input;-conda-lockdropped from the Renovate pattern, and a regex manager forconda-lock.ymlput back, each failtest_renovate_recompiles_the_conda_lock_tool_lock; the lock dropped from the pip-audit command failstest_pip_audit_audits_every_lock[requirements-conda-lock.lock.txt];--generate-hashesdropped from the header failstest_lock_is_compiled_with_hashes[requirements-conda-lock.lock.txt]..github/renovate.json5andci/README.md, so it is rebuilt onmainonce this merges.pip install --no-deps --require-hashes -rof it into a fresh Python 3.14 environment succeeds andimport rattler, yamlworks.Revert
A plain revert restores the unhashed
pip install "py-rattler ==0.28.1" "pyyaml ==6.0.3"line and the regex manager, and deletes the two requirements files. Nothing is stored.Checklist
pytestpassesruff,mypy, and schema checks passAGENTS.md.github/.licenserc.yaml)