Skip to content

Add test coverage for WorkflowRunner - #17

Open
Anilreddy2309 wants to merge 1 commit into
NVIDIA:mainfrom
Anilreddy2309:test/workflow-runner-coverage
Open

Add test coverage for WorkflowRunner#17
Anilreddy2309 wants to merge 1 commit into
NVIDIA:mainfrom
Anilreddy2309:test/workflow-runner-coverage

Conversation

@Anilreddy2309

Copy link
Copy Markdown

Summary

WorkflowRunner (nvflow/core/workflow_runner.py, ~500 lines) is the most complex piece of core orchestration logic in the repo — config loading with _base_ inheritance, dynamic stage expansion from models/checkpoints sections, dependency-closure validation, and stage execution — and had no dedicated test file. It was only exercised incidentally by tests written for unrelated features (tests/test_grounding_feature_gate.py, nvflow/recipes/multimodal/tests/test_hopchain_image_filter_paths.py), none of which touch run(), _expand_dynamic_stages(), _load_config_with_inheritance(), or the dependency-closure walk in _validate_stages().

What's covered

tests/test_workflow_runner.py (44 tests):

  • Config loading & _base_ inheritance — simple load, missing file, single-level and chained _base_ merging (child overrides base, non-overridden fields survive), missing base file, relative path resolution, OmegaConf interpolation resolution
  • Dynamic stage expansionmodels section expansion, workflow-level defaults forwarded but overridden by entry config, structural keys (recipe/cluster/etc.) not leaking into expanded stages, explicit stages: config not clobbered by expansion, checkpoints eval_steps (single int and list) expanding both stages and pipeline_stages
  • _get_expname, _resolve_env_names, _get_run_after_names — including per-environment dependency name expansion
  • _validate_stages / validate_config — valid stages pass, stage missing from pipeline config, unregistered stage, missing stage config block, transitive dependency-closure walk, dependency missing config block, dependency not registered, all 5 required top-level fields
  • run() — full pipeline execution order, subset runs, dependency filtering to only stages submitted this session, unknown-environment validation, the unregistered-sibling-stage preflight warning, fail-fast on an invalid requested stage before any stage executes

Design notes

Two construction strategies keep the suite fast and each fixture minimal: config-loading tests instantiate WorkflowRunner normally against real YAML files under tmp_path (since that I/O is what's being tested); everything downstream uses a _runner_from_config() helper that builds a WorkflowRunner from an in-memory dict via __new__ + the same post-load steps __init__ performs, skipping file I/O.

StageRegistry is a process-wide class-level registry, so tests that register stages use an isolated_registry fixture (monkeypatches StageRegistry._stages to a fresh dict for the test's duration) so registrations never leak into other tests or the real registry.

Test plan

  • ruff check / ruff format --check pass
  • mypy passes cleanly (one documented type: ignore[return-value] on a small typed accessor helper, explained inline — WorkflowRunner.config's declared type is a loose OmegaConf.to_container() union that mypy can't chain-index outside the constructor's own local flow; this is upstream typing looseness, not something this PR should try to fix by touching nvflow/core/workflow_runner.py's public API)
  • License header check passes
  • New suite passes (44/44) in an environment mirroring the exact CI install (pip install -e . --no-deps + tests/requirements-ci.txt, no nemo-skills/torch)
  • Full pytest tests/ suite passes (383 passed, 3 pre-existing skips) — no regressions

🤖 Generated with Claude Code

WorkflowRunner (nvflow/core/workflow_runner.py, ~500 lines) is the most
complex piece of core orchestration logic in the repo -- config loading
with _base_ inheritance, dynamic stage expansion from models/checkpoints
sections, dependency-closure validation, and stage execution -- and had
no dedicated test file. It was only exercised incidentally by tests
written for unrelated features (tests/test_grounding_feature_gate.py,
nvflow/recipes/multimodal/tests/test_hopchain_image_filter_paths.py),
none of which touch run(), _expand_dynamic_stages(),
_load_config_with_inheritance(), or the dependency-closure walk in
_validate_stages().

Adds tests/test_workflow_runner.py (44 tests) covering:

- Config loading & _base_ inheritance: simple load, missing file,
  single-level and chained _base_ merging (child overrides base,
  non-overridden fields survive), missing base file, relative path
  resolution, OmegaConf interpolation resolution
- Dynamic stage expansion: models section expansion, workflow-level
  defaults forwarded but overridden by entry config, structural keys
  (recipe/cluster/etc.) not leaking into expanded stages, explicit
  stages: config not clobbered by expansion, checkpoints eval_steps
  (single int and list) expanding both stages and pipeline_stages
- _get_expname, _resolve_env_names, _get_run_after_names (including
  per-environment dependency name expansion)
- _validate_stages / validate_config: valid stages pass, stage missing
  from pipeline config, unregistered stage, missing stage config block,
  transitive dependency-closure walk, dependency missing config block,
  dependency not registered, all 5 required top-level fields
- run(): full pipeline execution order, subset runs, dependency
  filtering to only stages submitted this session, unknown-environment
  validation, the unregistered-sibling-stage preflight warning,
  fail-fast on an invalid requested stage before any stage executes

Two construction strategies keep the suite fast and each fixture
minimal: config-loading tests instantiate WorkflowRunner normally
against real YAML files under tmp_path (since that I/O is what's being
tested); everything downstream uses a `_runner_from_config()` helper
that builds a WorkflowRunner from an in-memory dict via __new__ + the
same post-load steps __init__ performs, skipping file I/O.
StageRegistry is a process-wide class-level registry, so tests that
register stages use an `isolated_registry` fixture (monkeypatches
StageRegistry._stages to a fresh dict) so registrations never leak
into other tests or the real registry.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Anil Balireddy <anilbalireddi@gmail.com>
@Anilreddy2309
Anilreddy2309 requested a review from a team September 11, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant