Harden DownloadSecFilingsStage: validate_config + injection-safe command building - #14
Open
Anilreddy2309 wants to merge 1 commit into
Conversation
…and building DownloadSecFilingsStage.execute() had two gaps relative to the rest of the finance recipe's stages: 1. No validate_config() override, so a workflow YAML missing output_dir/sec_identity_email/sec_identity_company/tickers/ start_year/end_year raised a bare KeyError with no indication of which field or file to fix. Add validate_config() following the pattern used elsewhere in this recipe (e.g. train_validation_split.py), including the "config" file alt-path where tickers/start_year/end_year come from a separate YAML instead. 2. The shell command was built via raw f-string interpolation with manual double-quotes instead of nvflow.lib.cli_cmd.build_python_cmd, the shlex.quote-based helper this repo already uses for every other stage that submits a command this way (validate_questions, data_transformation, apply_prompt_template, convert_to_responses_api, prepare_data, prefetch_cache). A ticker or company name containing backticks or $(...) would not have been safely escaped under the old f-string approach. Migrate to build_python_cmd, which also standardizes on python3 (this repo's documented interpreter convention) instead of the unversioned python entry point the old code used. Add tests/test_download_sec_filings.py: validate_config coverage for every required field (direct-config and config-file-reference paths), plus execute() tests that stub nemo_skills.pipeline.cli in sys.modules (the module doesn't import nemo_skills at module level, so this works in the lightweight CI environment) to capture the exact rendered command -- including a regression test asserting that shell metacharacters in a ticker value are safely single-quoted rather than interpreted by the shell. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Anil Balireddy <anilbalireddi@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DownloadSecFilingsStage.execute()(nvflow/recipes/finance/stages/download/download_sec_filings.py) had two gaps relative to the rest of the finance recipe's stages:1. No
validate_config()override. A workflow YAML missingoutput_dir/sec_identity_email/sec_identity_company/tickers/start_year/end_yearraised a bareKeyErrorwith no indication of which field (or which YAML file) to fix. Addedvalidate_config()following the pattern already used elsewhere in this recipe (e.g.train_validation_split.py), including the alternate "config" file path wheretickers/start_year/end_yearcome from a separate referenced YAML instead of the stage config directly.2. Command built via raw f-string interpolation instead of the repo's own safe helper. The shell command was built with manual
f"..."string concatenation and hand-placed double quotes, instead ofnvflow.lib.cli_cmd.build_python_cmd— theshlex.quote-based helper this repo already uses for every other stage that submits a command this way (validate_questions,data_transformation,apply_prompt_template,convert_to_responses_api,prepare_data,prefetch_cache). A ticker or company name containing backticks or$(...)was not safely escaped under the old approach. Migrated tobuild_python_cmd, which also standardizes onpython3(this repo's documented interpreter convention pernvflow/lib/cli_cmd.py's own docstring) instead of the unversionedpythonentry point the old code used.Test plan
Added
tests/test_download_sec_filings.py:validate_configcoverage for every required field, both the direct-config and config-file-reference pathsexecute()tests that stubnemo_skills.pipeline.cliinsys.modules(the stage module doesn't importnemo_skillsat module level, only insideexecute(), so this works without the real dependency) to capture the exact rendered shell commandAAPL; rm -rf /,$(whoami)) are safely single-quoted in the rendered command rather than being interpretable by the shellVerified:
ruff check/ruff format --checkpassmypypasses on both filespip install -e . --no-deps+tests/requirements-ci.txt, nonemo-skills/torch)pytest tests/suite passes (351 passed, 3 pre-existing skips) — no regressions🤖 Generated with Claude Code