Add test coverage for WorkflowRunner - #17
Open
Anilreddy2309 wants to merge 1 commit into
Open
Conversation
WorkflowRunner (nvflow/core/workflow_runner.py, ~500 lines) is the most complex piece of core orchestration logic in the repo -- config loading with _base_ inheritance, dynamic stage expansion from models/checkpoints sections, dependency-closure validation, and stage execution -- and had no dedicated test file. It was only exercised incidentally by tests written for unrelated features (tests/test_grounding_feature_gate.py, nvflow/recipes/multimodal/tests/test_hopchain_image_filter_paths.py), none of which touch run(), _expand_dynamic_stages(), _load_config_with_inheritance(), or the dependency-closure walk in _validate_stages(). Adds tests/test_workflow_runner.py (44 tests) covering: - Config loading & _base_ inheritance: simple load, missing file, single-level and chained _base_ merging (child overrides base, non-overridden fields survive), missing base file, relative path resolution, OmegaConf interpolation resolution - Dynamic stage expansion: models section expansion, workflow-level defaults forwarded but overridden by entry config, structural keys (recipe/cluster/etc.) not leaking into expanded stages, explicit stages: config not clobbered by expansion, checkpoints eval_steps (single int and list) expanding both stages and pipeline_stages - _get_expname, _resolve_env_names, _get_run_after_names (including per-environment dependency name expansion) - _validate_stages / validate_config: valid stages pass, stage missing from pipeline config, unregistered stage, missing stage config block, transitive dependency-closure walk, dependency missing config block, dependency not registered, all 5 required top-level fields - run(): full pipeline execution order, subset runs, dependency filtering to only stages submitted this session, unknown-environment validation, the unregistered-sibling-stage preflight warning, fail-fast on an invalid requested stage before any stage executes Two construction strategies keep the suite fast and each fixture minimal: config-loading tests instantiate WorkflowRunner normally against real YAML files under tmp_path (since that I/O is what's being tested); everything downstream uses a `_runner_from_config()` helper that builds a WorkflowRunner from an in-memory dict via __new__ + the same post-load steps __init__ performs, skipping file I/O. StageRegistry is a process-wide class-level registry, so tests that register stages use an `isolated_registry` fixture (monkeypatches StageRegistry._stages to a fresh dict) so registrations never leak into other tests or the real registry. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Anil Balireddy <anilbalireddi@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
WorkflowRunner(nvflow/core/workflow_runner.py, ~500 lines) is the most complex piece of core orchestration logic in the repo — config loading with_base_inheritance, dynamic stage expansion frommodels/checkpointssections, dependency-closure validation, and stage execution — and had no dedicated test file. It was only exercised incidentally by tests written for unrelated features (tests/test_grounding_feature_gate.py,nvflow/recipes/multimodal/tests/test_hopchain_image_filter_paths.py), none of which touchrun(),_expand_dynamic_stages(),_load_config_with_inheritance(), or the dependency-closure walk in_validate_stages().What's covered
tests/test_workflow_runner.py(44 tests):_base_inheritance — simple load, missing file, single-level and chained_base_merging (child overrides base, non-overridden fields survive), missing base file, relative path resolution, OmegaConf interpolation resolutionmodelssection expansion, workflow-level defaults forwarded but overridden by entry config, structural keys (recipe/cluster/etc.) not leaking into expanded stages, explicitstages:config not clobbered by expansion,checkpointseval_steps(single int and list) expanding bothstagesandpipeline_stages_get_expname,_resolve_env_names,_get_run_after_names— including per-environment dependency name expansion_validate_stages/validate_config— valid stages pass, stage missing from pipeline config, unregistered stage, missing stage config block, transitive dependency-closure walk, dependency missing config block, dependency not registered, all 5 required top-level fieldsrun()— full pipeline execution order, subset runs, dependency filtering to only stages submitted this session, unknown-environment validation, the unregistered-sibling-stage preflight warning, fail-fast on an invalid requested stage before any stage executesDesign notes
Two construction strategies keep the suite fast and each fixture minimal: config-loading tests instantiate
WorkflowRunnernormally against real YAML files undertmp_path(since that I/O is what's being tested); everything downstream uses a_runner_from_config()helper that builds aWorkflowRunnerfrom an in-memory dict via__new__+ the same post-load steps__init__performs, skipping file I/O.StageRegistryis a process-wide class-level registry, so tests that register stages use anisolated_registryfixture (monkeypatchesStageRegistry._stagesto a fresh dict for the test's duration) so registrations never leak into other tests or the real registry.Test plan
ruff check/ruff format --checkpassmypypasses cleanly (one documentedtype: ignore[return-value]on a small typed accessor helper, explained inline —WorkflowRunner.config's declared type is a looseOmegaConf.to_container()union that mypy can't chain-index outside the constructor's own local flow; this is upstream typing looseness, not something this PR should try to fix by touchingnvflow/core/workflow_runner.py's public API)pip install -e . --no-deps+tests/requirements-ci.txt, nonemo-skills/torch)pytest tests/suite passes (383 passed, 3 pre-existing skips) — no regressions🤖 Generated with Claude Code