feat: pip-compile -> uv migration

[irfanuddinahmad]

Summary

Migrates edx-platform from pip-compile/pip-tools to pyproject.toml + uv, tracked in openedx/public-engineering#543.

Environments / install targets

Concept	Pre-migration	Post-migration
Production	make base-requirements -> pip-sync base.txt (= kernel.in + bundled.in, which pulls in github.in)	make base-requirements -> uv sync --no-default-groups --group bundled --frozen ([project.dependencies] merges kernel+github, and installs on every uv sync regardless of --group, so --group bundled alone reproduces the old base.txt)
Development	make dev-requirements -> pip-sync development.txt (= pip-tools.txt + testing.txt + doc.txt + assets.txt + dev-own deps)	make dev-requirements -> uv sync --group development --group ci --frozen (development itself includes testing+docs+assets). pip-tools itself is dropped, not needed under uv.
Testing (CI/tox)	make test-requirements -> pip-sync testing.txt (= base.txt + coverage.txt + testing-own deps, including tox)	make test-requirements -> uv sync --no-default-groups --group testing --frozen (testing includes coverage+bundled). Note: tox moved out of testing into the new ci group, so test-requirements no longer pulls in tox the way testing.txt used to -- intentional (this target runs pytest, not tox itself), but a real composition change, not just a rename.

Old file -> new equivalent

Old file	New equivalent
requirements/edx/kernel.in	[project.dependencies]
requirements/edx/github.in	absorbed into [project.dependencies] (merged with kernel.in)
requirements/edx/bundled.in	[dependency-groups].bundled
requirements/edx/testing.in	[dependency-groups].testing -- {bundled} + {coverage} + own deps, minus tox (see above)
requirements/edx/doc.in	[dependency-groups].docs -- {bundled} + own deps (doc.in did -r base.txt, i.e. kernel + bundled, and docs/conf.py needs bundled XBlocks like openedx-forum/ora2 importable to build at all)
requirements/edx/assets.in	[dependency-groups].assets
requirements/edx/development.in	[dependency-groups].development -- {testing}+{docs}+{assets} + own deps (pip-tools dropped)
requirements/edx/coverage.in	[dependency-groups].coverage
requirements/edx/semgrep.in	its own standalone sub-project, scripts/semgrep/{pyproject.toml,uv.lock} -- not a root dependency-group, see below
requirements/edx/openstack.txt	[project.optional-dependencies].openstack (a PEP 621 extra, not a [dependency-groups] entry -- installable via pip install openedx-platform[openstack])
requirements/constraints.txt + requirements/common_constraints.txt	[tool.edx_lint].uv_constraints (hand-maintained) + [tool.uv].constraint-dependencies (machine-generated from it)
requirements/pip-tools.{in,txt}	not needed -- pip-tools itself is gone
requirements/edx/private.in/private.txt	no direct successor file, but the same need (keeping extra personal packages installed across syncs) is covered by uv sync --inexact -- see requirements/README.rst for the exact workflow
requirements/edx/testing.in's tox entry	moved to [dependency-groups].ci, alongside new tox-uv (needed for tox.ini's uv-venv-lock-runner) -- make dev-requirements passes --group ci explicitly alongside --group development so tox stays available for local dev
(none -- new in this PR)	[dependency-groups].django42/django52: a real, independently-resolved Django-version test matrix, not a runtime patch on an already-resolved venv

requirements/edx/{base,assets,development}.txt and the 5 sub-projects' compat .txt files stay as machine-generated uv export outputs at their historical paths (for external tooling, e.g. Tutor's Dockerfile, that still installs from them directly) -- see the file-by-file breakdown below for exactly what each one exports.

File-by-file summary

Root packaging & build config

• pyproject.toml: core of the migration. Added the full [project.dependencies] list (previously just ["setuptools"]), a new [dependency-groups] tree (coverage, bundled, testing, docs, assets, development, django42, django52, ci), [tool.uv].default-groups/constraint-dependencies (machine-managed by edx_lint write_uv_constraints), [tool.edx_lint].uv_constraints (the hand-maintained version pins, each with a dated comment and issue link, carried over from the old constraints.txt), and a new [tool.coverage.*] tree.
  • django42/django52: a real, independently-resolved Django-version test matrix, not a runtime patch on an already-resolved venv -- [tool.uv].conflicts declares the two mutually exclusive so uv.lock carries a genuinely separate resolution for each (confirmed: django42 locks Django 4.2.30, django52/testing locks 5.2.x, with real transitive-dependency divergence between the two, not just a Django-version override).
  • [tool.uv].default-groups: an earlier version of this PR set this to ["development", "ci"] so a bare uv sync would install them, and briefly introduced a default group that just wrapped development+ci for default-groups to point at (removed as pure indirection -- default-groups accepts real group names directly). Both were reverted after landing: default-groups is now explicitly [] (uv's own built-in fallback is ["dev"], not "nothing", so this has to be set explicitly rather than omitted) because RTD's docs build has no way to opt out of default-groups and was silently installing the full dev/CI toolchain on every build as a result (see .readthedocs.yaml below). Every consumer that wants development+ci now passes --group development --group ci explicitly.
• scripts/update_uv_constraint_version.py (new): a helper the upgrade-one-python-dependency.yml workflow uses to bump a version pin in [tool.edx_lint].uv_constraints via a proper tomlkit TOML round-trip, instead of sed-patching a text file.
• .coveragerc (deleted): content moved verbatim into [tool.coverage.*] in pyproject.toml.
• Makefile: pre-requirements/pip-sync targets replaced with uv sync --group ... targets; compile-requirements rewritten to run edx_lint write_uv_constraints -> uv lock for the root project (and the same lock step for each of the 5 uv-managed sub-projects), then re-export compatibility .txt files at the old paths for external tools (e.g. Tutor's Dockerfile) that still pip install -r requirements/edx/base.txt directly. This export logic briefly lived in a separate scripts/compile_requirements_exports.py Python script, but per feanil's request it's back in the Makefile as inline shell -- visibility into the exact commands being run mattered more here than the script's structured-data abstraction (verified byte-identical output before switching either way).
• tox.ini: switched to runner = uv-venv-lock-runner + dependency_groups = testing; dropped quality from envlist (quality now runs via a dedicated CI workflow/Makefile target, not tox) and the now-redundant usedevelop/commands_pre = make test-requirements.
• package.json: compile-sass/compile-sass-dev now run uv run python scripts/compile_sass.py ... instead of the bare executable (which relied on its own #!/usr/bin/env python shebang finding the right Python on PATH) -- this way uv stays scoped to the actual Python invocation, and every caller of npm run build/build-dev (CI, scripts/watch_sass.sh, local dev) gets the right environment without needing to know to wrap the outer npm command itself.
• README.rst: install instructions updated from pip install -r requirements/edx/*.txt to uv sync --group ....

CI workflows

All follow the same mechanical pattern: drop the manual pip cache steps in favor of astral-sh/setup-uv's built-in cache (enable-cache: true), and replace every bare pip/pylint/mypy/./manage.py/etc. invocation with uv run <tool> -- not a $GITHUB_PATH splice. An earlier draft of this PR did splice .venv/bin onto $GITHUB_PATH job-wide instead of fixing individual invocations; that approach is gone everywhere now except one line in static-assets-check.yml that's unrelated to Python (node_modules/.bin, needed by npm-installed JS tooling).

• check-consistent-dependencies.yml: trigger-detection regex extended to also watch pyproject.toml/uv.lock/sub-project files, not just requirements/.
• check_python_dependencies.yml: re-enabled (was temporarily disabled earlier in this PR's history pending edx-repo-tools[find_dependencies] gaining the ability to scan uv.lock instead of .in/.txt files -- that upstream fix, openedx/repo-tools#735, has since merged). Now runs via uvx --from 'edx-repo-tools[find_dependencies]' find_python_dependencies --req-file uv.lock ..., no persistent install, no actions/setup-python.
• ci-static-analysis.yml, js-tests.yml, lint-imports.yml, migrations-check.yml, pylint-checks.yml, quality-checks.yml, semgrep.yml, unit-tests.yml: mechanical pip->uv swap as described above.
• compile-python-requirements.yml, upgrade-one-python-dependency.yml: updated to operate on pyproject.toml/uv.lock instead of .in/.txt/constraints.txt. The dependency-downgrade script now edits [tool.edx_lint].uv_constraints via a proper tomlkit TOML round-trip instead of sed-patching a text file. Both manually triggered and verified against this branch: compile-python-requirements ran make compile-requirements, detected no diff, and correctly exited without opening a PR; upgrade-one-python-dependency (package=django-filter) opened a PR with exactly the expected uv.lock + compat-export changes.
• static-assets-check.yml: same pip->uv swap, scoped the Python-deps-install step to uv sync --no-default-groups --group bundled --group assets --frozen (rather than pulling in the full development+ci set this job doesn't need). npm run build-dev/npm run build are called bare -- the package.json change above is what gets scripts/compile_sass.py the right Python, not this workflow.
• migrations-check.yml, unit-tests.yml: the non-pinned Django-version matrix entries now uv sync --group testing --group django$VERSION --frozen instead of bare-pip installing a different Django version on top of an already-resolved venv (which relied on pip check as the only safety net for anything transitively broken). This resolves the whole environment fresh for that Django line -- confirmed django-filter genuinely resolves differently under django42 (25.1) vs django52 (25.2), a real transitive divergence the old patch-on-top approach would have masked.
• units-test-scripts-structures-pruning.yml, units-test-scripts-user-retirement.yml: intentionally untouched -- they still pip install -r scripts/.../requirements/testing.txt, and those compatibility files continue to be regenerated (via uv export) at the same paths, so these two workflows keep working unmodified.

Documentation

• requirements/README.rst: rewritten to describe the new pyproject.toml/dependency-groups model instead of the old .in/.txt workflow, including how to replace the dropped private.in mechanism with uv sync --inexact, and how the 5 standalone sub-project directories (requirements/edx-sandbox, scripts/xblock, scripts/user_retirement, scripts/structures_pruning, scripts/semgrep) each carry their own independent pyproject.toml + uv.lock.
• requirements/edx-sandbox/README.rst: same treatment for the sandbox sub-project.
• scripts/semgrep/README.rst (new): explains why semgrep is isolated into its own sub-project rather than a root [dependency-groups] entry -- semgrep's own dependency chain (via wcmatch) is incompatible with other root-project dependencies (e.g. openedx-authz's pycasbin pin) when resolved together in one shared graph. master's old pip-compile-based setup avoided this by compiling semgrep.txt as a fully separate run; the root uv.lock needs the same isolation, which only a genuinely separate sub-project (like requirements/edx-sandbox/scripts/xblock already were) provides.
• docs/concepts/testing/testing.rst, docs/references/static-assets.rst: install instructions updated to the uv sync equivalents, scoped to the specific dependency-groups each workflow step actually needs.
• .readthedocs.yaml: replaced the old pip-requirements install with RTD's native uv support (python.install: [{method: uv, command: sync, groups: [docs]}]), reading directly from uv.lock -- this also installs the local package itself as an editable install automatically. groups: [docs] (not --only-group docs) so [project.dependencies] (Django/XBlock/etc.) resolves against uv.lock's pins instead of an unconstrained setuptools>=82 that breaks fs/pyfilesystem2's pkg_resources import; docs also directly includes the bundled group (matching doc.in's old -r base.txt), since docs/conf.py imports cms.envs.common, which needs bundled XBlocks importable to succeed. Fixed a real regression found after initial landing: RTD's groups: [docs] maps to uv sync --group docs, which -- with [tool.uv].default-groups previously set to ["development", "ci"] -- also implicitly installed the full dev/CI toolchain on every docs build (confirmed empirically: 401 packages vs. 335 for a docs-only install), and RTD's config has no way to pass --no-default-groups. Fixed at the source instead of working around it in RTD config: default-groups is now explicitly [], with every consumer that wants development+ci (make dev-requirements, the development.txt compat export) passing --group development --group ci explicitly. Verified against Read the Docs' actual build (not just locally) before landing, and re-verified after this fix.

Dependency files (requirements/, uv.lock, scripts/*)

• requirements/edx/*.in (all deleted): superseded by [project.dependencies]/[dependency-groups] in pyproject.toml.
• requirements/edx/base.txt, assets.txt, development.txt (kept, regenerated): machine-generated uv export compatibility exports at their historical paths for external tooling that still installs from them directly. base.txt exports [project.dependencies] plus the bundled group.
• requirements/edx/coverage.{in,txt}, doc.{in,txt}, testing.{in,txt}, semgrep.{in,txt}, bundled.in, github.in, kernel.in (all deleted): fully absorbed into pyproject.toml dependency-groups (or, for semgrep, its own standalone sub-project); no external tooling installed from these paths directly, so no compatibility export was needed.
• requirements/edx/openstack.txt (deleted): absorbed into [project.optional-dependencies].openstack -- a PEP 621 extra, not a [dependency-groups] entry.
• requirements/edx/private.readme (deleted): documented the private.in/private.txt mechanism, which itself was never checked into git (git-ignored). No direct successor file -- see the mapping table above for the uv sync --inexact replacement workflow.
• requirements/common_constraints.txt, requirements/constraints.txt (deleted): absorbed into [tool.uv].constraint-dependencies (machine-managed) and [tool.edx_lint].uv_constraints (hand-maintained) in pyproject.toml, with all original rationale comments and issue links preserved.
• requirements/pip-tools.{in,txt} (deleted): pip-tools itself is no longer needed.
• uv.lock, requirements/edx-sandbox/{pyproject.toml,uv.lock}, scripts/{xblock,user_retirement,structures_pruning,semgrep}/{pyproject.toml,uv.lock} (new): machine-generated lockfiles for the root project and its 5 independent standalone sub-projects (codejail sandbox, XBlock scripts, user-retirement scripts, structures-pruning scripts, semgrep). Each sub-project sets [tool.uv] package = false (they're script bags, not installable packages) and carries only the subset of root constraints relevant to its own deps.
• scripts/*/requirements/{base,testing}.txt (kept, regenerated): same compatibility-export treatment as requirements/edx/*.txt.

Misc scripts

• scripts/compile_sass.py: docstring rewritten with the minimal setup command needed to run just this script (uv sync --no-default-groups --only-group assets --no-install-project), verified to actually compile Sass correctly with that reduced environment.
• scripts/watch_sass.sh: one pip install -r ... message updated to uv sync --group development (the group that actually contains watchdog/watchmedo).
• scripts/xsslint_config.py: added .venv to the linter's skip-dirs list.

What's intentionally NOT done here (tracked externally)

• Tutor's Dockerfile installs from requirements/edx/{base,assets,development}.txt with plain pip. Those stay as uv export compatibility artifacts rather than being deleted, so no action is required on Tutor's side right now -- but Tutor maintainers should be aware these paths are now machine-generated, not hand-compiled. Verified tutor images build openedx-dev succeeds against this branch on tutor 22.0.0/Verawood (Python 3.12); the previous named release's Python 3.11 base can't satisfy this migration's requires-python = ">=3.12" at all, independent of anything in this PR.

Verification

• uv lock resolves cleanly (419 packages, root project); all 5 sub-projects (requirements/edx-sandbox, scripts/xblock, scripts/user_retirement, scripts/structures_pruning, scripts/semgrep) sync cleanly with uv sync --frozen.
• Re-ran make compile-requirements end-to-end for the root project and all 5 uv sub-projects -- regenerated compatibility export files are byte-identical to what's committed, confirming consistency.
• Manually triggered compile-python-requirements and upgrade-one-python-dependency (both workflow_dispatch-only, so they don't run automatically on this PR) against this branch -- see the CI workflows section above.
• All CI checks pass on this PR, including docs/readthedocs.org against the real Read the Docs build and tutor images build openedx-dev against a local Verawood (Python 3.12) environment.

---

🤖 Generated with Claude Code

[openedx-webhooks]

Thanks for the pull request, @irfanuddinahmad!

This repository is currently maintained by @openedx/wg-maintenance-openedx-platform-oncall.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

• If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
  • This process (including the steps you'll need to take) is documented here.
• If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

• Dependencies

  This PR must be merged before / after / at the same time as ...

• Blockers

  This PR is waiting for OEP-1234 to be accepted.

• Timeline information

  This PR must be merged by XX date because ...

• Partner information

  This is for a course on edx.org.

• Supporting documentation
• Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Details

---

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

• Overview of Review Process for Community Contributions
• Pull Request Status Guide
• Making changes to your pull request

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

• The size and impact of the changes that it introduces
• The need for product review
• Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

[irfanuddinahmad]

Re: the astral-sh/setup-uv SHA-pinning comments on unit-tests.yml, js-tests.yml, and lint-imports.yml —

Good catch, and worth noting: the org already ran a dedicated SHA-pinning sweep for this exact reason (openedx/.github#165, prompted by the tj-actions/changed-files supply-chain incident), but openedx-platform wasn't part of that ~121-repo effort.

That said, I don't think pinning just setup-uv here would meaningfully help: 82 other uses: refs in this repo's workflows (checkout, setup-python, setup-node, etc.) are still floating tags, so the actual attack surface stays open regardless, and pinning only one action creates an inconsistent special case for little real benefit. I'd rather this be a proper repo-wide pinact-based migration (same approach used for the other 121 repos) as its own dedicated follow-up, not a partial fix bundled into this PR.

[salman2013]

Minor naming consistency issue: the group names aren't uniform. You have doc (abbreviated) alongside development (full word)

[irfanuddinahmad]

Fair point — renamed doc to docs to match development's un-abbreviated naming (and the docs/ folder). Updated its one internal reference plus .readthedocs.yaml, which also referenced it by name. Fixed in 6d4d18a.

[salman2013]

We still have both development and dev groups. I think having both is a bit confusing. Could we use more consistent naming?

[irfanuddinahmad]

Good point -- renamed the dev group to default (added explicit tool.uv.default-groups = ["default"] to preserve bare uv sync's current behavior, since that was relying on uv's built-in "dev" default-group name). Now only development remains, wrapped by default = development + ci. Fixed in a1d3ea1.

[irfanuddinahmad]

Fixed -- consolidated to just development, no more separate dev group.

[farhan]

@irfanuddinahmad @salman2013

doc.in has been moved to docs altering the name not in unparity with master
Headup on it if it's under consideration

[irfanuddinahmad]

Already decided, not overlooked -- see salman2013's thread above (3656743218): renamed doc->docs on 2026-07-27 to match development's un-abbreviated style.

[salman2013]

@irfanuddinahmad This commit changes are not related to PR so either they raised in conflicts resolution or AI tried to fix something by own.

[salman2013]

why we are using pip here?

[irfanuddinahmad]

Fixed -- this now uses uv sync --no-default-groups --only-group coverage --no-install-project --frozen, picking up the coverage dependency-group already defined in pyproject.toml for exactly this purpose. Restores the pinning that got lost when this was a bare pip install. See 1c592ea.

[farhan]

Requested a few nits.

The PR looks to be in good shape now. 👏👏👏

💡 Major thing to discuss is what strategy do we follow for operators to migrate to uv from old requirements.

There's a lot of extra work happening because we're trying to keep the old requirements files around but I think we can drop a lot of that if we can do a fast-track DEPR before we merge this work. As long as operators know what to do instead of what they were doing, they can adapt with sufficient warning.

source comment

[irfanuddinahmad]

Re: the fast-track DEPR suggestion -- openedx/public-engineering#552 already covers this (filed 2026-07-22). But the transition can't actually start without this PR.
master's pyproject.toml is still a stub with no uv.lock, so there's nothing yet for Tutor / edx-enterprise / openedx-translations / repo-tools to switch to. All 4 consumers tracked in #552's task list are still unmigrated.

[irfanuddinahmad]

Recent testing on this branch, for the record:

• [tool.uv].constraint-dependencies removed setuptools<82 and re-locked -- uv immediately jumped to 84.0.0 Restored, re-locked clean.
• make upgrade verified end-to-end: real package bumps across root + all 5 sub-projects, all 9 compat .txt exports correctly scoped (verified base.txt excludes pytest/mypy/tox, development.txt includes both), idempotent on a second run, reverted after.
• make compile-requirements is a true no-op on a clean checkout post-Makefile-reversion -- byte-identical hashes on all 16 relevant files.
• scripts/structures_pruning/README.md's "without uv" fallback (pip install + virtualenv) verified working standalone in a real throwaway venv, both the quick-start and testing paths.
• tutor images build openedx-dev: compile-sass/compile-sass-dev failed inside the Docker image because bare uv run ignores $VIRTUAL_ENV and looks for a project-local .venv, which the image never creates (it installs into /openedx/venv via uv pip install instead). Fixed with uv run --active (verified safe for both cases), re-ran the full image build -- confirmed compile-sass now completes cleanly.

[farhan]

2 nits and a query, rest all seems good 👏 👏 👏
Kudos on the retesting

👍 from my side

Let @feanil have its final review and merge it.

[feanil]

Looks like this needs to be rebased but then I think it's good to merge.

[irfanuddinahmad]

Thanks. Opened #39023 to drop it.