Repository navigation
test: wait for the performance phase before ^C in sigint tests - #559
Merged
arekay-nv merged 3 commits intoNov 10, 2026
Merged
Conversation
test_sigint_grace_expiry_abandons_wedged_drain waited for metrics/.ready and then slept a fixed 3 s before SIGSTOPping the aggregator. .ready precedes HTTP worker startup and the session's STARTED, and the aggregator publishes no live snapshot until STARTED. When worker startup outlasted the sleep, the wedged-then-SIGKILLed aggregator left neither final_snapshot.json nor a live frame, so no result_summary.json was written. Wait for the "Starting phase: performance" log line (logged right after STARTED) before the sleep, in this test and in test_group_sigint_exits_gracefully, which shares the pattern. Capture the run log in the grace-expiry test and print its tail if the summary is missing. Reuse the new _wait_for_log helper in the drain-window test, and note in BenchmarkSession._run_phase that tests key on this log line. Closes mlcommons#558 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #559 +/- ##
=======================================
Coverage ? 82.63%
=======================================
Files ? 170
Lines ? 24331
Branches ? 0
=======================================
Hits ? 20106
Misses ? 4225
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
6 of 10 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes the intermittent CI failure in
test_sigint.py::test_sigint_grace_expiry_abandons_wedged_drain(
FileNotFoundError: .../performance/result_summary.json), seen on 8 runsacross 4 branches; see #558.
Cause: the test waited for
metrics/.readyand then slept a fixed 3 sbefore SIGSTOPping the aggregator and sending ^C.
.readyprecedes HTTPworker startup (
_create_issuer) and the session'sSTARTED, and theaggregator publishes no live snapshot until
STARTED. When worker startupoutlasted the sleep, the wedged-then-SIGKILLed aggregator left neither a
final_snapshot.jsonnor a live frame, sodrain_and_build_reporthadnothing to build
result_summary.jsonfrom. The product behaviour is withinthe module's contract ("or no artifacts at all"); the test's timing
assumption was wrong.
Changes
tests/integration/commands/test_sigint.py_wait_for_log()helper and_PERF_PHASE_STARTEDconstant(
"Starting phase: performance", logged byBenchmarkSession._run_phaseright after
STARTEDis published).test_sigint_grace_expiry_abandons_wedged_drain: captures the run log,waits for the phase-start line before the 3 s sleep (now ~12 aggregator
publish ticks past
STARTED), and prints the log tail ifresult_summary.jsonis missing.test_group_sigint_exits_gracefully: same wait, since it uses the same.ready+ fixed-sleep pattern and expects interrupted perf artifacts.test_sigint_in_drain_window_keeps_summary_authoritative: its inlinelog-polling loop now uses
_wait_for_log(same needle and timeout)._wait_services_readydocstring, which claimed issuingstarts right after service readiness.
src/inference_endpoint/load_generator/session.py: comment-only. Notesthat
test_sigint.pywaits on theStarting phase:log line andtest_benchmark_command.pyasserts on it, so the wording must stay insync.
Type of change
Related issues
Closes #558
Testing
These tests are Linux-only (
/proc, CPU affinity), so they were run in aLinux container (
ghcr.io/astral-sh/uv:python3.12-bookworm-slim, aarch64).Limiting it to 0.5 CPU slows worker startup the way a loaded runner does,
which reproduces the flake:
test_sigint_grace_expiry_abandons_wedged_drain@ 0.5 CPUmainFileNotFoundErrorFileNotFoundErrortest_sigint.pywith no CPU limit: 6 passed, 3 of 3 runs.-m "not slow and not performance") on macOS:2599 passed, 55 failed. The failing set is identical to
main's; all55 are macOS-only (CPU affinity /
/proc).os.sched_setaffinityerrors in untouched files.Known, unchanged: at 0.5 CPU,
test_sigint_in_drain_window_keeps_summary_authoritativefails on bothmainand this PR (nofinal_snapshot.json). It is not seen in CI; detailsare in the #558 comment.
Checklist
🤖 Generated with Claude Code