Skip to content

chore(ci): decouple python deps from docker images#1380

Merged
spomichter merged 2 commits intodevfrom
chore/ci-decouple-deps
Mar 1, 2026
Merged

chore(ci): decouple python deps from docker images#1380
spomichter merged 2 commits intodevfrom
chore/ci-decouple-deps

Conversation

@spomichter
Copy link
Contributor

Problem

python packages are baked into docker images at build time. any pyproject.toml change triggers a full rebuild cascade before tests can run. this burns ~1 hour of CI time unnecessarily.

Solution

moved python dep installation from dockerfile build to test runtime via uv sync. now pyproject changes skip image rebuilds and install deps at test time instead (~30s vs ~1h).

changes:

  • docker/python/Dockerfile: removed COPY, uv pip install, pydrake stub deletion
  • docker/dev/Dockerfile: removed uv pip install .[dev]
  • docker.yml: removed pyproject.toml from python path filter, renamed run-ros-tests → tests / run-mypy → mypy, simplified test trigger conditions to always run when workflow fires
  • tests.yml: added uv sync --all-extras --frozen step and pydrake stub deletion before test execution

Breaking Changes

none

How to Test

  1. merge this PR to dev
  2. rebuild images once (docker workflow will trigger)
  3. make a pyproject.toml change (add a dep or bump version)
  4. observe that docker build is skipped and tests run directly

expected: tests install deps via uv sync (~30s), no image rebuild

Contributor License Agreement

  • I have read and approved the CLA

@greptile-apps
Copy link
Contributor

greptile-apps bot commented Feb 28, 2026

Greptile Summary

decouples python dependency installation from docker image builds by removing pyproject.toml from the python path filter and moving uv pip install commands from Dockerfiles to a runtime uv sync step in tests.yml. when pyproject.toml changes, tests now install dependencies at runtime (~30s) instead of triggering full image rebuild cascades (~1h).

key changes:

  • removed pyproject.toml from python path filter (addresses previous review feedback)
  • python and dev Dockerfiles no longer copy project files or install dependencies
  • tests.yml now runs uv sync --all-extras --frozen before executing tests
  • test jobs simplified to always run when workflow fires (previously conditional on specific path changes)

architecture shift:

  • before: dependencies baked into images → rebuild on pyproject.toml changes → ~1h CI time
  • after: base images without deps → install deps at test time → ~30s overhead per test run

the change from installing specific extras to --all-extras may install additional dependency groups (cuda, psql, agents, etc.) not previously included, potentially affecting installation time.

Confidence Score: 4/5

  • Safe to merge - clean architecture refactoring with clear performance benefits
  • The PR successfully addresses its goal of decoupling python dependencies from docker builds. The previous review concern about pyproject.toml in the path filter has been resolved. The logic is sound: dependencies removed from Dockerfiles, runtime installation added to tests, and workflow triggers updated appropriately. Minor deduction for the change from specific extras to --all-extras which may install more dependencies than needed and potentially affect the claimed ~30s installation time.
  • tests.yml - verify that --all-extras installation time remains within acceptable limits after merge

Important Files Changed

Filename Overview
.github/workflows/docker.yml removed pyproject.toml from python path filter (enables dep changes without image rebuilds), simplified test job conditions to always run, renamed test jobs for clarity
.github/workflows/tests.yml added runtime dependency installation via uv sync --all-extras --frozen and pydrake stub cleanup (previously in Dockerfiles) - installs more extras than before
docker/python/Dockerfile removed project copy, dependency installation, and pydrake stub deletion - dependencies now installed at test runtime instead
docker/dev/Dockerfile removed uv pip install .[dev] step - dev dependencies now installed at test runtime via uv sync

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[pyproject.toml change] --> B{Before this PR}
    A --> C{After this PR}
    
    B --> D[Triggers python path filter]
    D --> E[Rebuild python image<br/>uv pip install deps]
    E --> F[Rebuild ros-python image]
    F --> G[Rebuild dev image<br/>uv pip install dev]
    G --> H[Rebuild ros-dev image]
    H --> I[Run tests<br/>~1h total]
    
    C --> J[No path filter match]
    J --> K[Skip all image rebuilds]
    K --> L[Tests run with existing image]
    L --> M[uv sync --all-extras<br/>installs deps at runtime]
    M --> N[Run tests<br/>~30s overhead]
    
    style E fill:#ffcccc
    style F fill:#ffcccc
    style G fill:#ffcccc
    style H fill:#ffcccc
    style I fill:#ffcccc
    style M fill:#ccffcc
    style N fill:#ccffcc
Loading

Last reviewed commit: 732c15d

Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

4 files reviewed, 2 comments

Edit Code Review Agent Settings | Greptile

@spomichter
Copy link
Contributor Author

@greptile check now

git config --global --add safe.directory '*'

- name: Install Python dependencies
run: uv sync --all-extras --frozen
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--all-extras installs more dependencies than the old Dockerfiles ([misc,cpu,sim,drone,unitree,web,perception,visualization,manipulation,dev]). Now includes cuda, psql, dds, docker, base, agents etc. Consider using uv sync --extra misc --extra cpu ... to match the original scope if installation time exceeds ~30s.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

@spomichter spomichter force-pushed the chore/ci-decouple-deps branch 3 times, most recently from e953c1c to a4de37d Compare February 28, 2026 17:25
moved python dep installation from docker build to test runtime.
pyproject.toml changes no longer trigger image rebuilds.

- docker/python/Dockerfile: removed COPY, uv pip install, pydrake stub deletion
- docker/dev/Dockerfile: removed uv pip install .[dev]
- docker.yml: removed pyproject.toml from python path filter, renamed job to run-tests/run-mypy
- tests.yml: added uv sync --all-extras --no-extra dds --frozen, activate venv before running tests
@spomichter spomichter force-pushed the chore/ci-decouple-deps branch from a4de37d to e0bda65 Compare March 1, 2026 22:25
open3d leak fixed in #1386, workaround no longer needed.
keeps venv activation since deps are now installed at test time.
@spomichter spomichter merged commit 4f58f74 into dev Mar 1, 2026
10 checks passed
@spomichter spomichter deleted the chore/ci-decouple-deps branch March 1, 2026 23:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant