Skip to content

test: wait for the performance phase before ^C in sigint tests - #559

Merged
arekay-nv merged 3 commits into
mlcommons:mainfrom
arekay-nv:test/558-sigint-wait-for-phase-start
Nov 10, 2026
Merged

arekay-nv merged 3 commits into
mlcommons:mainfrom
arekay-nv:test/558-sigint-wait-for-phase-start

Conversation

@arekay-nv

Copy link
Copy Markdown
Collaborator

What does this PR do?

Fixes the intermittent CI failure in
test_sigint.py::test_sigint_grace_expiry_abandons_wedged_drain
(FileNotFoundError: .../performance/result_summary.json), seen on 8 runs
across 4 branches; see #558.

Cause: the test waited for metrics/.ready and then slept a fixed 3 s
before SIGSTOPping the aggregator and sending ^C. .ready precedes HTTP
worker startup (_create_issuer) and the session's STARTED, and the
aggregator publishes no live snapshot until STARTED. When worker startup
outlasted the sleep, the wedged-then-SIGKILLed aggregator left neither a
final_snapshot.json nor a live frame, so drain_and_build_report had
nothing to build result_summary.json from. The product behaviour is within
the module's contract ("or no artifacts at all"); the test's timing
assumption was wrong.

Changes

  • tests/integration/commands/test_sigint.py
    • New _wait_for_log() helper and _PERF_PHASE_STARTED constant
      ("Starting phase: performance", logged by BenchmarkSession._run_phase
      right after STARTED is published).
    • test_sigint_grace_expiry_abandons_wedged_drain: captures the run log,
      waits for the phase-start line before the 3 s sleep (now ~12 aggregator
      publish ticks past STARTED), and prints the log tail if
      result_summary.json is missing.
    • test_group_sigint_exits_gracefully: same wait, since it uses the same
      .ready + fixed-sleep pattern and expects interrupted perf artifacts.
    • test_sigint_in_drain_window_keeps_summary_authoritative: its inline
      log-polling loop now uses _wait_for_log (same needle and timeout).
    • Corrected the _wait_services_ready docstring, which claimed issuing
      starts right after service readiness.
  • src/inference_endpoint/load_generator/session.py: comment-only. Notes
    that test_sigint.py waits on the Starting phase: log line and
    test_benchmark_command.py asserts on it, so the wording must stay in
    sync.

Type of change

  • Bug fix

Related issues

Closes #558

Testing

These tests are Linux-only (/proc, CPU affinity), so they were run in a
Linux container (ghcr.io/astral-sh/uv:python3.12-bookworm-slim, aarch64).
Limiting it to 0.5 CPU slows worker startup the way a loaded runner does,
which reproduces the flake:

test_sigint_grace_expiry_abandons_wedged_drain @ 0.5 CPU Run 1 Run 2 Run 3
main fail: abort took 6.2 s fail: FileNotFoundError fail: FileNotFoundError
This PR pass pass pass
  • Full test_sigint.py with no CPU limit: 6 passed, 3 of 3 runs.
  • CI's test selection (-m "not slow and not performance") on macOS:
    2599 passed, 55 failed. The failing set is identical to main's; all
    55 are macOS-only (CPU affinity / /proc).
  • Pre-commit: all hooks pass except mypy, which on macOS reports the known
    os.sched_setaffinity errors in untouched files.

Known, unchanged: at 0.5 CPU,
test_sigint_in_drain_window_keeps_summary_authoritative fails on both
main and this PR (no final_snapshot.json). It is not seen in CI; details
are in the #558 comment.

Checklist

  • Code follows project style
  • Pre-commit hooks pass
  • Documentation updated (if needed)

🤖 Generated with Claude Code

test_sigint_grace_expiry_abandons_wedged_drain waited for metrics/.ready
and then slept a fixed 3 s before SIGSTOPping the aggregator. .ready
precedes HTTP worker startup and the session's STARTED, and the
aggregator publishes no live snapshot until STARTED. When worker startup
outlasted the sleep, the wedged-then-SIGKILLed aggregator left neither
final_snapshot.json nor a live frame, so no result_summary.json was
written.

Wait for the "Starting phase: performance" log line (logged right after
STARTED) before the sleep, in this test and in
test_group_sigint_exits_gracefully, which shares the pattern. Capture the
run log in the grace-expiry test and print its tail if the summary is
missing. Reuse the new _wait_for_log helper in the drain-window test, and
note in BenchmarkSession._run_phase that tests key on this log line.

Closes mlcommons#558

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@arekay-nv
arekay-nv requested a review from a team November 3, 2026 04:00
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

@github-actions github-actions Bot added the size/normal PR Review Policy: <=500 non-test lines & <=20 files label Nov 3, 2026
@codecov-commenter

codecov-commenter commented Nov 3, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@520eb4d). Learn more about missing BASE report.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #559   +/-   ##
=======================================
  Coverage        ?   82.63%           
=======================================
  Files           ?      170           
  Lines           ?    24331           
  Branches        ?        0           
=======================================
  Hits            ?    20106           
  Misses          ?     4225           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@arekay-nv
arekay-nv merged commit 63210f3 into mlcommons:main Nov 10, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/normal PR Review Policy: <=500 non-test lines & <=20 files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: flaky test_sigint_grace_expiry_abandons_wedged_drain - SIGINT can land before the session STARTs

2 participants