[CRCR] Treat expected non-success outcomes (xfail/xcancel/xtimeout) as successes in pass rate

[subinz1]

Summary

Fixes the CRCR pass rate calculation to correctly handle expected non-success outcomes in health-probe repos like pytorch/crcr-test.

The crcr-test repo uses jobs with x-prefixed names (xfail, xcancel, xtimeout) that intentionally end in failure, cancellation, or timeout to test the relay's handling of these outcomes (see pytorch/crcr-test#19). These expected results were being counted as failures in the ClickHouse pass rate queries, causing pytorch/crcr-test to show a degraded health status even though the relay is functioning correctly.

Changes

Three ClickHouse queries updated with the same logic:

1. crcr_summary — CRCR summary page (/crcr)
2. crcr_success_rate — CRCR metrics page (/crcr/metrics)
3. crcr_backend_summary — per-repo dashboard (/crcr/[org]/[repo])

For each query:

• Successes: now includes jobs where job_name LIKE '%xfail%', '%xcancel%', or '%xtimeout%'
• Failures: excludes xfail jobs
• Timed out: excludes xtimeout jobs

This uses the job_name convention established in pytorch/crcr-test#19 — jobs prefixed with x are intentional/expected outcomes, not bugs.

Fixes #8306

Test plan

• Verify CRCR summary page shows improved pass rate for pytorch/crcr-test
• Verify metrics page charts reflect corrected rates
• Verify per-repo dashboard for pytorch/crcr-test shows accurate stats

[vercel]

@subinz1 is attempting to deploy a commit to the Meta Open Source Team on Vercel.

A member of the Team first needs to authorize it.

[subinz1]

Dependency: This PR depends on pytorch/crcr-test#19 being merged first — that PR introduces the xfail/xcancel/xtimeout job naming convention that this PR relies on for the pass rate fix.

[atalman]

HI @subinz1 I don't see any difference between currently deployed prod change and this one. Still showing 85% success and degraded state same as https://hud.pytorch.org/crcr/pytorch/crcr-test:

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
torchci	Ready	Preview	Aug 6, 2026 2:50pm

[huydhn]

I wonder if we can set the conclusion of xfail/xcancel/xtimeout to something like neutral to distinguish them https://docs.github.com/en/webhooks/webhook-events-and-payloads#workflow_job

[atalman]

Right problem to fix — crcr-test shouldn't show degraded health for outcomes it produces on purpose. Two issues to sort out before this leaves draft.

1. The name check isn't paired with a conclusion check

The SQL counts a job as a pass if it's named xfail/xcancel/xtimeout, without checking what it actually did. The intended rule is "named xfail and it failed" — that AND is missing.

This matters more than a normal counting bug: crcr-test exists to prove the relay correctly carries failure/cancel/timeout conclusions. With this change those three jobs report green no matter what happens to them, including if the relay stops propagating conclusions entirely. An xfail that unexpectedly passes also reads as green. The probe can no longer go red, which is the one thing it's for.

2. Double counting

successes admits all three names, but failures only excludes xfail and timed_out only excludes xtimeout. So an xtimeout job that fails is counted as both a success and a failure. Same for xcancel that times out, and xfail that times out. pass_rate and the timeout rate can sum past 100%.

Suggested shape

Define the predicate once, pairing each name with its expected conclusion, and apply it symmetrically. That makes the buckets mutually exclusive by construction and fixes both issues:

-- is_expected
{repo: String} = 'pytorch/crcr-test' AND (
       (job_name LIKE '%xfail%'    AND conclusion = 'failure')
    OR (job_name LIKE '%xcancel%'  AND conclusion = 'cancelled')
    OR (job_name LIKE '%xtimeout%' AND conclusion = 'timed_out')
)

successes = countIf(conclusion = 'success'   OR  is_expected)
failures  = countIf(conclusion = 'failure'   AND NOT is_expected)
timed_out = countIf(conclusion = 'timed_out' AND NOT is_expected)

On normalizing at the source instead

There's been discussion of having the crcr-test callback report Failed/Cancelled as Pass rather than filtering in SQL. Two reasons I don't think that works here:

• It can't cover timeouts. A timed-out job never calls back — the relay synthesizes the conclusion itself in cleanup_handler._finalize_timed_out_check_run when the in-progress record expires, precisely because the downstream stopped reporting. So xtimeout would still need a SQL special case, leaving us half source-side and half query-side.
• It removes the signal. If the downstream reports success for a job designed to fail, the relay only ever carries success conclusions, and we stop exercising the failure/cancel transport end to end.

Longer term the cleanest version is probably an explicit expected_conclusion column on crcr_workflow_job populated by the relay, so consumers just compare conclusion = expected_conclusion — no per-query convention, no hardcoded repo name, and an xfail that passes is naturally a failure. The predicate above is a fine interim.

Merge order

This touches crcr_backend_summary/query.sql, which #8421 also rewrites (replacing flaky_jobs with timeout_rate = timed_out / total_jobs), and #8425 is stacked on that. Worth picking an order — and after both land, note that timed_out would exclude xtimeout jobs while total_jobs still counts them.

Minor, non-blocking: 'pytorch/crcr-test' is hardcoded six times across the three files; crcr_backend_summary uses {repo: String} while the other two use downstream_repo; and the explanatory comment is only in crcr_summary.

[subinz1]

@atalman Great catch on both issues — the missing conclusion check and the double-counting are real bugs.

I've reworked the queries to use the paired is_expected predicate you suggested:

-- Expected probe outcomes: name must match its intended conclusion
is_expected = downstream_repo = 'pytorch/crcr-test' AND (
    (job_name LIKE '%xfail%'    AND conclusion = 'failure')
    OR (job_name LIKE '%xcancel%'  AND conclusion = 'cancelled')
    OR (job_name LIKE '%xtimeout%' AND conclusion = 'timed_out')
)

Applied symmetrically:

• successes = countIf(conclusion = 'success' OR is_expected)
• failures = countIf(conclusion = 'failure' AND NOT is_expected)
• timed_out = countIf(conclusion = 'timed_out' AND NOT is_expected)

This fixes both issues:

1. An xfail that unexpectedly succeeds now shows as an anomaly (not silently green)
2. Buckets are mutually exclusive — no double counting

Also noted the merge order concern with #8421 and #8425. Will rebase after those land.