Skip to content

fix(intelligence): require a meaningful change before reporting latency and volume anomalies - #324

Merged
vishr merged 1 commit into
mainfrom
fix/detector-effect-size
Oct 16, 2026
Merged

vishr merged 1 commit into
mainfrom
fix/detector-effect-size

Conversation

@vishr

@vishr vishr commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

Summary

Latency and volume anomalies fired on the z-score alone. The baseline is the scatter of a few five-minute buckets, so a steady service has a tiny standard deviation, and a 0.1 ms or 5% shift scored as a severe anomaly. One hour of the OpenTelemetry Demo produced 36 findings across 18 services. Examples:

  • ad P95 latency: 1.4ms (baseline 1.3ms, z-score 3.15)
  • payment volume: 263 spans (baseline 250, z-score 3.52)

As a result, the dashboard's anomaly overlay covered every panel for the whole hour.

Both checks now also need practical significance, in the style of the existing error-rate floor (minErrorRateStddev):

  • Latency: p95 must rise by at least 20% and at least 5 ms. A 1 ms floor applies to the baseline scatter.
  • Volume: the count must change by at least 25% in either direction. A one-span floor applies to the scatter. A service present in the baseline but silent now is evaluated, so traffic stopping is reported.

Only qualified findings affect severity, insights and the health score. No storage, API or format changes.

Evidence

Replayed 60 detector evaluations over one hour of an OpenTelemetry Demo snapshot (rollups rebuilt with the repository's refresh code):

Kind / severity Findings before → after Episodes before → after
Latency / warn 54 → 2 8 → 1
Latency / bad 17 → 0 5 → 0
Volume / warn 8 → 0 5 → 0
Volume / bad 8 → 0 5 → 0
Total 87 → 2 23 → 1

The one remaining episode is a frontend-web p95 rise from 202 to 243 ms.

New tests cover:

  • the noisy examples above, which no longer fire;
  • regressions that still fire: 20 → 60 ms p95, a service going silent, traffic doubling, a 30% drop;
  • the boundary at each constant.

Verification

  • go test ./internal/intelligence/... ./internal/alert/... ./internal/annotations/...: passed.
  • just check (pre-push): passed.

…cy and volume anomalies

Latency and volume anomalies fired on the z-score alone. The baseline is the
scatter of a few five-minute buckets, so a steady service has a tiny standard
deviation and a 0.1 ms or 5% shift scored as a severe anomaly. One hour of
the OpenTelemetry Demo produced 36 findings across 18 services, and the
dashboard's anomaly overlay covered every panel for the whole hour.

Both checks now also need practical significance, in the style of the
existing error-rate floor:

- latency: p95 must rise by at least 20% and at least 5 ms, with a 1 ms
  floor on the baseline scatter;
- volume: the count must change by at least 25% in either direction, with a
  one-span floor on the scatter. A service present in the baseline but
  silent now is evaluated, so traffic stopping is reported.

Only qualified findings affect severity, insights and the health score.
Replaying the detector over one hour of a Demo snapshot reduced findings
from 87 to 2; the remaining one is a frontend-web p95 rise from 202 to 243 ms.

Claude-Session: https://claude.ai/code/session_01XA52nWyQ3R5f5kK8kjpAQW
@vishr
vishr force-pushed the fix/detector-effect-size branch from 95acb6d to ad7cb2c Compare October 16, 2026 20:15
@vishr
vishr merged commit 35044ca into main Oct 16, 2026
14 checks passed
@vishr
vishr deleted the fix/detector-effect-size branch October 16, 2026 20:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant