fix(intelligence): score error-rate spikes against the rate's own variance - #265
Merged
Merged
Conversation
…iance
The z-score divides a difference of rates, so its denominator has to be
the scatter of the rate from bucket to bucket. It was
STDDEV(CASE WHEN status IN ('STATUS_CODE_ERROR','ERROR') THEN 1.0 ELSE 0.0 END)
over raw spans, which is the scatter of individual span outcomes --
sqrt(p(1-p)), about 0.34 at a 14% error rate. Those are different
quantities, and the mismatch punished exactly the services worth
watching: the noisier the service, the larger the denominator. At the
13.8% baseline the live demo actually ran, clearing the 2.0 threshold
required the rate to jump 69 percentage points. A tripling from 13.8%
to 41% scored 0.80.
The denominator is now the standard deviation of the per-bucket error
rate across the baseline window, and both windows average over the same
5-minute buckets -- the same asymmetry the volume detector had.
It is also floored, because a healthy service's error rate is flat at
zero and its bucket-to-bucket stddev is therefore exactly zero. Dividing
by that hit the CASE's 0.0 fallback, so the service that had just
started failing was the one that could not alert: going from no errors
to 42% scored 0.00. One percentage point is the least noise worth
assuming. It keeps a single stray error in three thousand spans at
z=0.03 while a real break reaches z=42.
Measured on the fixtures in error_rate_sql_test.go, before -> after:
13.8% -> 41% (noisy baseline) 0.80 -> 27.40 now fires
13.8% -> 13.8% (noisy baseline) 0.01 -> 0.20 stays quiet
0% -> 42% (flat baseline) 0.00 -> 42.33 now fires
0% -> 0.03% (one stray error) 0.00 -> 0.03 stays quiet
Verified on the demo across five detector cycles: 1-2 anomalies each,
health 85-95, no errors. The concern that a more sensitive denominator
would produce an alert flood does not survive contact with the data.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #264, which fixed the volume detector and named this as the
remaining reason error-rate detection was desensitised.
The z-score divides a difference of rates, so its denominator has to be
the scatter of the rate from bucket to bucket. It was
STDDEVof aper-span 0/1 indicator -- the scatter of individual span outcomes,
sqrt(p(1-p)), about 0.34 at a 14% error rate. Different quantities,and the mismatch punished the services most worth watching: the noisier
the service, the larger the denominator. At the demo's 13.8% baseline,
clearing the 2.0 threshold needed a 69 percentage point jump.
It is also floored at one percentage point, because a healthy service's
error rate is flat at zero, giving it a bucket-to-bucket stddev of
exactly zero -- so the service that had just started failing was the one
that could not alert.
Measured on the fixtures, before -> after:
Both directions are asserted, because a test that only checks the quiet
cases passes against a detector that never fires at all.
Verified on the demo across five detector cycles: 1-2 anomalies each,
health 85-95, no errors.