Repository navigation
testing doctest - #2652
Draft
grzegorz-k-karch wants to merge 2 commits into
Draft
testing doctest#2652grzegorz-k-karch wants to merge 2 commits into
grzegorz-k-karch wants to merge 2 commits into
Conversation
Signed-off-by: Grzegorz Karch <gkarch@nvidia.com>
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueNo actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configuration
📒 Files selected for processing (9)
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review. 📝 WalkthroughWalkthroughThe change adds executable Markdown scenarios and a runner, then uses them to test the LLM QAT README with reduced-scale fixtures. It also updates example-test change detection and adds documentation and unit tests for the scenario framework. ChangesExecutable Markdown documentation tests
Priority: ➖ Normal Estimated code review effort: 3 (Moderate) | ~25 minutes Change: Other Sequence Diagram(s)sequenceDiagram
participant pytest
participant parse_markdown
participant run_scenario
participant scenario_process
participant prepare_llm_qat_workspace
participant verify_llm_qat_workspace
pytest->>parse_markdown: collect README scenarios
pytest->>run_scenario: execute selected scenario
run_scenario->>scenario_process: launch with timeout
scenario_process->>prepare_llm_qat_workspace: create temporary workspace
scenario_process->>scenario_process: run README shell commands
scenario_process->>verify_llm_qat_workspace: check QAT artifacts
scenario_process-->>run_scenario: return output or failure
run_scenario-->>pytest: return output or raise error
Suggested reviewers: Merge Risk: ⚪ Minimal · up to This change adds opt-in documentation tests and a CI trigger for README changes. It does not alter production behavior, and no concrete merge-blocking risk was found in the supplied context. 🚥 Pre-merge checks | ✅ 5 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (5 passed)
Full details: Docstring CoverageExplanation Docstring coverage is 21.05% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 19 functions across 5 files. (4 skipped: 4 unsupported.) ✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2652 +/- ##
==========================================
- Coverage 69.49% 68.07% -1.43%
==========================================
Files 614 620 +6
Lines 68631 71158 +2527
==========================================
+ Hits 47694 48438 +744
- Misses 20937 22720 +1783
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Signed-off-by: Grzegorz Karch <gkarch@nvidia.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
!! Draft description below, not yet reviewed by PR author, so don't bother reviewing it yourself
What does this PR do?
Type of change: New tests
Adds an opt-in pytest framework that executes reader-visible Markdown shell fences, helping catch documentation regressions without maintaining separate copies of example commands.
Hidden HTML comments define scenarios, resource profiles, timeouts, Python setup and artifact verification. The runner preserves shell state between fences, captures output, reports Markdown file/line locations, and cleans up subprocess groups. Unmarked fences never execute.
The pilot covers recipe listing and the PTQ → QAT → export workflow in
examples/llm_qat/README.md, using a tiny local Qwen3 checkpoint and synthetic data while retaining real NVFP4 quantization and FSDP2 training. It also adds the missing working-directory change and ensures README changes trigger the existing example CI lane.Usage
From the repository root:
See
tests/_test_utils/doc_tests/README.mdfor the annotation contract and container setup.Testing
git diff --checkpassed.GPU validation used the PyTorch 26.07 container’s matching Torch/FlashAttention binaries in a temporary environment. The existing development virtual environment had an incompatible FlashAttention binary.
Before your PR is "Ready for review"
Make sure you read and follow Contributor guidelines and your commits are signed (
git commit -s -S).Make sure you read and follow the Security Best Practices.
CONTRIBUTING.md: N/A — no copied third-party code or new project dependency./claude review.Additional Information
The smoke tests establish that the documented commands compose and produce real artifacts. They do not establish full-model accuracy, convergence, memory requirements, serving compatibility or inference performance.
Installation and QAD fences remain outside the pilot. Existing backend tests are retained for broader coverage, including DDP and DeepSpeed. Annotated Markdown executes as trusted repository test code; the runner is not a security sandbox.
Summary by CodeRabbit
New Features
Documentation
Tests