You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(health): fail TransferBench healthcheck test when TransferBench reports a failure - #688
test_transfer_bench_healthcheck only ran scan_test_results, whose patterns ("test FAIL ", "ABORT", ...) do not match TransferBench's healthcheck lines such as "Testing all-to-all XGMI copies ........FAIL (56 test(s))".
TransferBench has non-0 exit code was not checked either, and the Run Deck capture is best-effort, so a failing healthcheck was reported as a pytest pass.
Run the healthcheck with orch.exec(detailed=True) and fail the test on a non-0 exit code.
The healthcheck preset exits non-0 when any sub-test fails or when it cannot run
(for example on an unsupported GPU model).
Failed sub-tests are named from the "Testing ... FAIL (N test(s))" lines.
We have seen on MI300X with TransferBench v1.67.00 and v1.71.00 that all-to-all XGMI copies failed on 56 of 56 pairs
(exit code 2, also through the sudo -n bash -c wrapper), however the CVS test passed.
…eports a failure
test_transfer_bench_healthcheck only ran scan_test_results, whose patterns
("test FAIL ", "ABORT", ...) do not match TransferBench's healthcheck verdict
lines such as "Testing all-to-all XGMI copies ........FAIL (56 test(s))".
TransferBench's non-zero exit code was not checked either, and the Run Deck
capture is best-effort, so a failing healthcheck was reported as a pytest pass.
Run the healthcheck with orch.exec(detailed=True) and fail the test on a
non-zero exit code. The healthcheck preset exits non-zero when any sub-test
fails or when it cannot run (for example on an unsupported GPU model). Failed
sub-tests are named from the "Testing ... FAIL (N test(s))" lines.
Seen on MI300X with TransferBench v1.67.00 and v1.71.00: all-to-all XGMI
copies failed on 56 of 56 pairs (exit code 2, also through the
"sudo -n bash -c" wrapper), but the CVS test passed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This PR updates CVS’s TransferBench healthcheck to detect failures that the generic output scanner misses, using command exit codes and named sub-test failures.
Changes:
Request detailed execution results and validate healthcheck outcomes.
Add unit tests for passing, failing, and verbose output, plus detailed-result forwarding.
Document the corrected failure behavior in the changelog.
File
Description
cvs/tests/health/transferbench_cvs.py
Collect detailed results and check healthcheck failures.
cvs/lib/unittests/test_transferbench_health.py
Add healthcheck parsing and execution-option coverage.
…esults
Address review feedback on the TransferBench healthcheck exit-code check:
- Pass print_console=False when run_transferbench requests detailed
results. With SSH sharding (more than CVS_HOSTS_PER_SHARD hosts), the
merged-output logger calls .splitlines() on each result and fails on the
{output, exit_code} dicts. The healthcheck test logs the output itself
through print_test_output.
- Fail every expected host (orch.hosts) that has no result. Sharded runs
send commands only to reachable hosts, so a host pruned during an earlier
probe was absent from the results and the healthcheck passed without
running there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add integration test for healthcheck failure propagation
cvs/tests/health/transferbench_cvs.py:566
Please add an integration-level unit test that calls test_transfer_bench_healthcheck with a mocked orchestrator and checks the actual pytest outcome for both zero and non-zero exits. The new tests mock fail_test and exercise the parser and command helper separately, so they do not verify that this call propagates a recorded failure through update_test_result; a wiring regression could restore the original false pass without failing them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test_transfer_bench_healthcheckonly ranscan_test_results, whose patterns("test FAIL ", "ABORT", ...)do not match TransferBench'shealthchecklines such as"Testing all-to-all XGMI copies ........FAIL (56 test(s))".TransferBench has non-0 exit code was not checked either, and the Run Deck capture is best-effort, so a failing
healthcheckwas reported as apytestpass.Run the
healthcheckwithorch.exec(detailed=True)and fail the test on a non-0 exit code.The
healthcheckpreset exits non-0 when any sub-test fails or when it cannot run(for example on an unsupported GPU model).
Failed sub-tests are named from the
"Testing ... FAIL (N test(s))"lines.We have seen on MI300X with TransferBench v1.67.00 and v1.71.00 that all-to-all XGMI copies failed on 56 of 56 pairs
(exit code 2, also through the
sudo -n bash -cwrapper), however the CVS test passed.Motivation
Technical Details
Test Plan
Test Result
Submission Checklist