From 5c040c1528563efd7a6c4bbbea5ea3283fdd69a7 Mon Sep 17 00:00:00 2001 From: Tyler Beebe Date: Fri, 24 Jul 2026 14:44:26 -0400 Subject: [PATCH 1/2] =?UTF-8?q?docs:=20correct=20Flag=20as=20Flaky=20prece?= =?UTF-8?q?dence=20=E2=80=94=20broken=20status=20overrides=20the=20manual?= =?UTF-8?q?=20flag?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The flag-as-flaky page claimed a flagged test is marked flaky regardless of what monitors report. In reality status resolution picks the most severe classification, so an active broken monitor keeps the test Broken while flagged. Document the precedence, the disabled flag button while a test is broken, and steer users toward muting or reconfiguring the responsible monitor when they believe a classification is incorrect. --- flaky-tests/detection/flag-as-flaky.mdx | 30 +++++++++++-------- flaky-tests/detection/index.mdx | 1 + .../managing-detected-flaky-tests.mdx | 2 +- 3 files changed, 20 insertions(+), 13 deletions(-) diff --git a/flaky-tests/detection/flag-as-flaky.mdx b/flaky-tests/detection/flag-as-flaky.mdx index a0704757..260d11a7 100644 --- a/flaky-tests/detection/flag-as-flaky.mdx +++ b/flaky-tests/detection/flag-as-flaky.mdx @@ -3,7 +3,7 @@ title: "Flag as Flaky" description: "Manually mark a test as flaky from the test detail page" og:title: "Flagging tests as flaky in Trunk" --- -Manually mark a test as flaky when you know it's unreliable but automated monitors haven't detected it yet — or when you want to override the system's assessment. +Manually mark a test as flaky when you know it's unreliable but automated monitors haven't detected it yet — or when you want to keep a test classified as flaky while monitors consider it healthy. ## When to Use It @@ -22,10 +22,14 @@ Manually mark a test as flaky when you know it's unreliable but automated monito Once flagged: -- The test is immediately marked as **flaky**, regardless of what automated monitors report. +- The test is immediately marked as **flaky**, even if no automated monitor has detected it. - An amber banner appears below the header showing who flagged it, when, and the reason (if provided). - The flag is additive — if automated monitors later detect the test as flaky too, both signals coexist. + +A manual flag does not override a **broken** status. Broken always takes precedence over flaky, so if an active monitor classifies the test as broken, the test shows as broken until that monitor resolves or is [muted](./index#muting-monitors). While a test is broken, the **Flag as Flaky** button is disabled — if you believe the broken classification is incorrect, [mute the monitor](./index#muting-monitors) or update its configuration instead. You can still remove an existing flag while the test is broken. + + ### Removing the Flag 1. On the test detail page, find the amber "Manually flagged as flaky" banner. @@ -39,16 +43,18 @@ After removing: ## Relationship to Monitors -The "Flag as Flaky" action is independent of automated monitors (threshold-based, pass-on-retry). It does not appear in the Monitors tab. - -| Scenario | Test status | -| ----------------------------------- | ------------------------ | -| No monitors active, no flag | Healthy | -| Monitors active, no flag | Flaky (detected) | -| No monitors active, flag set | Flaky (manually flagged) | -| Monitors active, flag set | Flaky (both) | -| Flag removed, monitors still active | Flaky (detected) | -| Flag removed, monitors inactive | Healthy | +The "Flag as Flaky" action is separate from automated monitors (threshold-based, pass-on-retry) and does not appear in the Monitors tab, but it participates in the same [status resolution](./index#how-monitors-work) as health classification monitors: the most severe status wins, and broken outranks flaky. If a test's status looks wrong — for example, it shows as broken when you believe it's merely flaky — [mute](./index#muting-monitors) or reconfigure the monitor responsible rather than reaching for the flag. + +| Scenario | Test status | +| --------------------------------------------- | -------------------------------- | +| No monitors active, no flag | Healthy | +| Flaky monitors active, no flag | Flaky (detected) | +| No monitors active, flag set | Flaky (manually flagged) | +| Flaky monitors active, flag set | Flaky (both) | +| Broken monitor active, flag set | Broken (broken takes precedence) | +| Broken monitor resolves or is muted, flag set | Flaky (manually flagged) | +| Flag removed, flaky monitors still active | Flaky (detected) | +| Flag removed, monitors inactive | Healthy | ## Flag History diff --git a/flaky-tests/detection/index.mdx b/flaky-tests/detection/index.mdx index a6fa7f38..fde972aa 100644 --- a/flaky-tests/detection/index.mdx +++ b/flaky-tests/detection/index.mdx @@ -121,6 +121,7 @@ You can also unmute a monitor early from the test case view. | --- | --- | | Fix is in progress and you want to suppress noise temporarily | **Mute** the monitor for a few days | | Test is flaky but no automated monitor has caught it | Use [**Flag as Flaky**](./flag-as-flaky) to mark it as flaky | +| A test shows as **Broken** (or Flaky) and you believe the classification is wrong | **Mute** the monitor that's driving the status, or update its configuration (thresholds, branch scope) so it stops misfiring — a [Flag as Flaky](./flag-as-flaky) can't override a broken status | | You want to stop a monitor from evaluating a test permanently | Adjust the monitor's branch scope or thresholds instead | | You want to suppress all flaky signals for a test | Mute each active monitor individually, or address the root cause | diff --git a/flaky-tests/management/managing-detected-flaky-tests.mdx b/flaky-tests/management/managing-detected-flaky-tests.mdx index de08af61..2ff0f63a 100644 --- a/flaky-tests/management/managing-detected-flaky-tests.mdx +++ b/flaky-tests/management/managing-detected-flaky-tests.mdx @@ -83,7 +83,7 @@ Muting suppresses the monitor's contribution to the test's status. If the muted ### Step 5: Flag flaky tests -If automated detection hasn't caught a test you know is flaky, you can manually [flag it as flaky](../detection/flag-as-flaky) from the test detail page. Flagged tests are treated as flaky regardless of automated detection state, and the flag can be removed at any time. +If automated detection hasn't caught a test you know is flaky, you can manually [flag it as flaky](../detection/flag-as-flaky) from the test detail page. Flagged tests are treated as flaky even when no monitor detects them, and the flag can be removed at any time. The one exception is a **Broken** status: broken takes precedence over flaky, so a flag can't override an active broken monitor. If you believe a broken classification is incorrect, mute that monitor or update its configuration instead. ### Step 6: Quarantine flaky tests From 17e9296fa08601aa1f96f05a5aedc8431d4d14d2 Mon Sep 17 00:00:00 2001 From: Tyler Beebe Date: Fri, 24 Jul 2026 14:57:31 -0400 Subject: [PATCH 2/2] Update flaky-tests/management/managing-detected-flaky-tests.mdx Co-authored-by: Tyler Jang --- flaky-tests/management/managing-detected-flaky-tests.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flaky-tests/management/managing-detected-flaky-tests.mdx b/flaky-tests/management/managing-detected-flaky-tests.mdx index 2ff0f63a..ffb553a0 100644 --- a/flaky-tests/management/managing-detected-flaky-tests.mdx +++ b/flaky-tests/management/managing-detected-flaky-tests.mdx @@ -83,7 +83,7 @@ Muting suppresses the monitor's contribution to the test's status. If the muted ### Step 5: Flag flaky tests -If automated detection hasn't caught a test you know is flaky, you can manually [flag it as flaky](../detection/flag-as-flaky) from the test detail page. Flagged tests are treated as flaky even when no monitor detects them, and the flag can be removed at any time. The one exception is a **Broken** status: broken takes precedence over flaky, so a flag can't override an active broken monitor. If you believe a broken classification is incorrect, mute that monitor or update its configuration instead. +If automated detection hasn't caught a test you know is flaky, you can manually [flag it as flaky](../detection/flag-as-flaky) from the test detail page. Flagged tests are treated as flaky even when no monitor detects them, and the flag can be removed at any time. Note that active broken monitors will take precedence over the manual flaky flag. ### Step 6: Quarantine flaky tests