Skip to content

Commit db708a8

Browse files
authored
fix: Color service map nodes by absolute error-rate thresholds (#3125)
## Why Service map nodes were always some shade of red, and you couldn't tell a service with zero errors from one with a trace of them. Two causes, both in `getNodeColors`: 1. **The ramp was max-normalized.** `intensity = value / max` meant the darkest node was always whichever service had the highest error rate *on the current graph* — whether that was 0.3% or 60%. A perfectly healthy map painted its best-of-a-good-bunch service full dark red. 2. **Zero sat at the bottom of the red ramp,** rendering `hsl(0 35% 90%)` — a pale pink. A service at 5% of the graph max landed at `hsl(0 37% 88%)`, a 2-point lightness difference on a 32px circle. Not perceptible. Worst case: when no service has errors at all, `metricMax.errorRate === 0`, so every node renders identically pale pink — indistinguishable from "everyone has a trace of errors." This is out of step with how other tools handle it. Datadog and New Relic leave healthy nodes neutral and only color on an absolute threshold breach (New Relic explicitly won't draw green for an operational node). AWS X-Ray and Grafana's Tempo service graph use a proportional donut. Nobody asks users to read error rate off a lightness ramp. ## What changed Error rate now maps to four **absolute** states: | Error rate | Fill | |---|---| | `0` | `hsl(220 8% 82%)` — neutral cool grey | | `0 – 1%` | `hsl(0 48% 73%)` | | `1 – 5%` | `hsl(0 61% 56%)` | | `≥ 5%` | `hsl(0 72% 42%)` | - The neutral is a **hue shift, not desaturation**. At 32px a cool grey reads as a different kind of thing; a washed-out red just reads as a little red. - The three red stops come from the existing ramp, so no new hexes enter the palette. - 5% is the threshold `ServiceMapTooltip` already used for its alarm color, now exported as `ERROR_RATE_HIGH` and shared, so the fill and the hover text can't drift apart. - **Latency and throughput are unchanged** — they keep the max-normalized ramp. Neither has a universal threshold to anchor to. - The legend gradient emits hard stops for error rate (four discrete steps rather than a blend, since a continuous bar would imply precision the bucketing doesn't have), and labels the top of the scale `≥5%` instead of the observed max. No changes to `ServiceMapNode`, `ServiceMap.tsx`, or the SCSS — the node already just consumes `{ backgroundColor, borderColor }`. ## Trade-off With hard buckets a service at 4.9% and one at 60% paint identically. A proportional donut would keep that distinction, but it's a much larger change (and a four-way success/4xx/5xx/throttle split isn't available to us — the source schema only has `statusCodeExpression`, and OTel's `StatusCode` is just `Ok`/`Error`/`Unset`). The tooltip shows the exact percentage on hover, so the detail is recoverable. Thresholds are hardcoded as exported constants. A service on a 99.9% SLO and a batch job have different notions of "elevated" — if that comes up, per-source overrides are the natural extension. ## Testing `packages/app/src/components/ServiceMap/__tests__/utils.test.ts` — existing ramp tests re-pointed at `throughput` (still max-normalized), plus a new block covering the absolute buckets: zero renders neutral, buckets ignore `max`, thresholds are inclusive lower bounds, and the neutral state still draws a darker border. Full app suite green (223 suites / 3664 tests), `make ci-lint` clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
1 parent e31e5d8 commit db708a8

6 files changed

Lines changed: 267 additions & 34 deletions

File tree

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
---
2+
'@hyperdx/app': patch
3+
---
4+
5+
Color service map nodes by absolute error-rate thresholds instead of scaling
6+
against the worst service on the graph. A service with no errors now renders
7+
neutral grey rather than a pale red, and the remaining nodes fall into three
8+
fixed buckets (under 1%, 1-5%, and 5% or above). A service the map has no error
9+
data for — one that only calls others, with no incoming requests in the window —
10+
renders hollow rather than filled, so "nothing measured" no longer looks like
11+
"nothing wrong".
12+
13+
Previously every node was a shade of red whose intensity was normalized against
14+
the graph-wide maximum, so a map whose worst service sat at 0.3% errors painted
15+
it the same deep red as one at 60%, and a healthy service was indistinguishable
16+
from one with a trace of errors. The legend for error rate now shows the four
17+
discrete steps, marks the 1% and 5% boundaries, and adds a key for the hollow
18+
state. Latency and throughput coloring is unchanged.

‎packages/app/src/components/ServiceMap/ServiceMapLegend.tsx‎

Lines changed: 23 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@ import { formatDurationMs } from '@/utils';
55

66
import type { ServiceMapMetricMax } from './ServiceMapMetricContext';
77
import {
8+
ERROR_RATE_ELEVATED,
9+
ERROR_RATE_HIGH,
810
formatRate,
911
getMetricGradientCss,
1012
getRequestsPerSecond,
@@ -16,9 +18,9 @@ import {
1618
import styles from './ServiceMap.module.scss';
1719

1820
/**
19-
* Formats the graph-wide max for the active metric into a human label so the
20-
* relative (max-normalized) color ramp is interpretable — e.g. the fully
21-
* saturated end of the scale corresponds to this value.
21+
* Labels the top of the active metric's color scale. Latency and throughput
22+
* are normalized against the graph, so their label is the graph-wide max;
23+
* error rate is bucketed on absolute thresholds and ignores `max`.
2224
*/
2325
function formatMax(
2426
metric: ServiceMapMetric,
@@ -27,14 +29,16 @@ function formatMax(
2729
dateRange: [Date, Date],
2830
isSingleTrace?: boolean,
2931
): string {
30-
// Only treat missing/invalid data as "n/a"; a real zero (e.g. a map with no
31-
// errors) is meaningful and should render as a formatted 0 for its metric.
32+
// Only treat missing/invalid data as "n/a"; a real zero is meaningful and
33+
// should render as a formatted 0 for its metric.
3234
if (!Number.isFinite(max) || max < 0) {
3335
return 'n/a';
3436
}
3537
switch (metric) {
3638
case 'errorRate':
37-
return max === 0 ? '0%' : `${max.toFixed(max < 10 ? 1 : 0)}%`;
39+
// Error rate is bucketed on absolute thresholds rather than normalized
40+
// against the graph, so the scale ends at the top bucket, not at `max`.
41+
return `≥${ERROR_RATE_HIGH}%`;
3842
case 'latency':
3943
// A zero latency max is a "no latency data" sentinel (see
4044
// getServiceMetricValue), not a genuine 0ms, so surface it as n/a.
@@ -81,12 +85,24 @@ export default function ServiceMapLegend({
8185
/>
8286
<Group justify="space-between" gap="xs" wrap="nowrap">
8387
<Text size="xxs" c="var(--color-text-muted)">
84-
low
88+
{metric === 'errorRate' ? 'none' : 'low'}
8589
</Text>
90+
{/* Lands on the right stop only while the bar has four equal
91+
segments, which puts the 1% threshold at its midpoint. */}
92+
{metric === 'errorRate' && (
93+
<Text size="xxs" c="var(--color-text-muted)">
94+
{ERROR_RATE_ELEVATED}%
95+
</Text>
96+
)}
8697
<Text size="xxs" c="var(--color-text)">
8798
{formatMax(metric, max, source, dateRange, isSingleTrace)}
8899
</Text>
89100
</Group>
101+
{metric === 'errorRate' && (
102+
<Text size="xxs" c="var(--color-text-muted)">
103+
Hollow = no data
104+
</Text>
105+
)}
90106
<Text size="xxs" c="var(--color-text-muted)">
91107
Node size = throughput
92108
</Text>

‎packages/app/src/components/ServiceMap/ServiceMapNode.tsx‎

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -55,12 +55,14 @@ export default function ServiceMapNode(
5555

5656
const { metric, metricMax } = use(ServiceMapMetricContext);
5757

58-
const { backgroundColor, borderColor } = getNodeColors(
59-
getServiceMetricValue(data, metric),
60-
metricMax[metric],
61-
props.selected,
62-
metric,
63-
);
58+
const { backgroundColor, borderColor, borderStyle, borderWidth } =
59+
getNodeColors(
60+
getServiceMetricValue(data, metric),
61+
metricMax[metric],
62+
props.selected,
63+
metric,
64+
totalIncomingRequestCount > 0,
65+
);
6466

6567
// Fallback matches the schema default (3 = ms); in practice the field is
6668
// always present on a parsed source.
@@ -116,6 +118,8 @@ export default function ServiceMapNode(
116118
style={{
117119
backgroundColor,
118120
borderColor,
121+
borderStyle,
122+
borderWidth,
119123
width: size,
120124
height: size,
121125
}}

‎packages/app/src/components/ServiceMap/ServiceMapTooltip.tsx‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ import {
1313
import { formatDurationMs } from '@/utils';
1414

1515
import {
16+
ERROR_RATE_HIGH,
1617
formatApproximateNumber,
1718
formatRate,
1819
navigateToTraceSearch,
@@ -56,7 +57,7 @@ export default function ServiceMapTooltip({
5657
// Only alarm (red) once errors are non-trivial; a fraction of a percent reads
5758
// as amber and a clean service stays neutral-danger-free.
5859
const errorColor =
59-
errorPercentage >= 5
60+
errorPercentage >= ERROR_RATE_HIGH
6061
? 'var(--color-text-danger)'
6162
: 'var(--color-chart-warning)';
6263

‎packages/app/src/components/ServiceMap/__tests__/utils.test.ts‎

Lines changed: 134 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ import { SourceKind, TTraceSource } from '@hyperdx/common-utils/dist/types';
33

44
import {
55
deriveDisplayMetrics,
6+
ERROR_RATE_HIGH,
67
formatApproximateNumber,
78
formatRate,
89
getMetricGradientCss,
@@ -254,38 +255,134 @@ function parseHsl(color: string | undefined): {
254255
}
255256

256257
describe('getNodeColors', () => {
258+
// Latency and throughput stay max-normalized; error rate is bucketed on
259+
// absolute thresholds and is covered separately below.
257260
describe('sequential ramp', () => {
258261
it('goes from a light tint at zero intensity to a dark shade at max', () => {
259-
const low = parseHsl(getNodeColors(0, 20, false).backgroundColor);
260-
const high = parseHsl(getNodeColors(20, 20, false).backgroundColor);
262+
const low = parseHsl(
263+
getNodeColors(0, 20, false, 'throughput').backgroundColor,
264+
);
265+
const high = parseHsl(
266+
getNodeColors(20, 20, false, 'throughput').backgroundColor,
267+
);
261268
// Light -> dark: lightness decreases, saturation increases as the metric
262269
// value climbs (a proper sequential scale, not a grey->color ramp).
263270
expect(high.l).toBeLessThan(low.l);
264271
expect(high.s).toBeGreaterThan(low.s);
265272
});
266273

267274
it('increases intensity monotonically with the value', () => {
268-
const l0 = parseHsl(getNodeColors(0, 20, false).backgroundColor).l;
269-
const l5 = parseHsl(getNodeColors(5, 20, false).backgroundColor).l;
270-
const l10 = parseHsl(getNodeColors(10, 20, false).backgroundColor).l;
271-
const l20 = parseHsl(getNodeColors(20, 20, false).backgroundColor).l;
272-
expect(l0).toBeGreaterThanOrEqual(l5);
273-
expect(l5).toBeGreaterThanOrEqual(l10);
274-
expect(l10).toBeGreaterThanOrEqual(l20);
275+
const lightnessAt = (value: number) =>
276+
parseHsl(getNodeColors(value, 20, false, 'throughput').backgroundColor)
277+
.l;
278+
expect(lightnessAt(0)).toBeGreaterThanOrEqual(lightnessAt(5));
279+
expect(lightnessAt(5)).toBeGreaterThanOrEqual(lightnessAt(10));
280+
expect(lightnessAt(10)).toBeGreaterThanOrEqual(lightnessAt(20));
275281
});
276282

277283
it('caps at the max value even when the value is higher', () => {
278-
expect(getNodeColors(30, 20, false)).toEqual(
279-
getNodeColors(20, 20, false),
284+
expect(getNodeColors(30, 20, false, 'throughput')).toEqual(
285+
getNodeColors(20, 20, false, 'throughput'),
280286
);
281287
});
282288

283289
it('treats a non-positive max as zero intensity', () => {
284-
expect(getNodeColors(5, 0, false)).toEqual(getNodeColors(0, 20, false));
290+
expect(getNodeColors(5, 0, false, 'throughput')).toEqual(
291+
getNodeColors(0, 20, false, 'throughput'),
292+
);
285293
});
286294

287295
it('renders the border a fixed step darker than the fill', () => {
288-
const { backgroundColor, borderColor } = getNodeColors(10, 20, false);
296+
const { backgroundColor, borderColor } = getNodeColors(
297+
10,
298+
20,
299+
false,
300+
'throughput',
301+
);
302+
expect(parseHsl(borderColor).l).toBeLessThan(parseHsl(backgroundColor).l);
303+
expect(parseHsl(borderColor).h).toBe(parseHsl(backgroundColor).h);
304+
});
305+
});
306+
307+
describe('absolute error-rate buckets', () => {
308+
const fillAt = (errorPercentage: number, max = 100) =>
309+
parseHsl(
310+
getNodeColors(errorPercentage, max, false, 'errorRate').backgroundColor,
311+
);
312+
313+
it('paints zero errors a neutral color, not the low end of the red ramp', () => {
314+
const none = fillAt(0);
315+
const lowest = fillAt(0.1);
316+
expect(none.h).not.toBe(SERVICE_MAP_METRIC_HUE.errorRate);
317+
expect(none.s).toBeLessThan(lowest.s);
318+
expect(none).not.toEqual(lowest);
319+
});
320+
321+
it('keeps zero neutral no matter what the graph-wide max is', () => {
322+
expect(fillAt(0, 0)).toEqual(fillAt(0, 100));
323+
});
324+
325+
it('ignores the graph-wide max so a shade always means the same rate', () => {
326+
// The bug this replaces: a 0.3%-error service painted full-intensity red
327+
// just because it was the worst on the graph.
328+
expect(fillAt(0.3, 0.3)).toEqual(fillAt(0.3, 100));
329+
expect(fillAt(0.3, 0.3)).not.toEqual(fillAt(60, 100));
330+
});
331+
332+
it('darkens across the low, elevated and high buckets', () => {
333+
expect(fillAt(0.5).l).toBeGreaterThan(fillAt(3).l);
334+
expect(fillAt(3).l).toBeGreaterThan(fillAt(20).l);
335+
});
336+
337+
it('treats each threshold as an inclusive lower bound', () => {
338+
expect(fillAt(0.99)).toEqual(fillAt(0.01));
339+
expect(fillAt(1)).toEqual(fillAt(4.99));
340+
expect(fillAt(1)).not.toEqual(fillAt(0.99));
341+
expect(fillAt(ERROR_RATE_HIGH)).toEqual(fillAt(100));
342+
expect(fillAt(ERROR_RATE_HIGH)).not.toEqual(fillAt(4.99));
343+
});
344+
345+
it('renders a service with no measured requests outline-only', () => {
346+
// A caller-only service has errorPercentage 0 but no error data at all;
347+
// a solid neutral would read as a clean record.
348+
const noData = getNodeColors(0, 100, false, 'errorRate', false);
349+
expect(noData.backgroundColor).toBe('transparent');
350+
expect(noData.borderColor).toBe('var(--color-chart-gray)');
351+
});
352+
353+
it('separates no-data from no-errors by border style, not colour', () => {
354+
// chart-gray sits within two points of the neutral node's derived
355+
// border, so the dash is what actually distinguishes the two states.
356+
const noData = getNodeColors(0, 100, false, 'errorRate', false);
357+
const noErrors = getNodeColors(0, 100, false, 'errorRate');
358+
expect(noData.borderStyle).toBe('dashed');
359+
expect(noErrors.borderStyle).toBe('solid');
360+
});
361+
362+
it('keeps a selected no-data node visible on a white canvas', () => {
363+
// The usual white selection ring would vanish: the fill is transparent
364+
// and the light-mode canvas is --color-bg-body, which is white. Width
365+
// carries the selection instead.
366+
const selected = getNodeColors(0, 100, true, 'errorRate', false);
367+
expect(selected.borderColor).not.toBe('white');
368+
expect(selected.borderWidth).toBeGreaterThan(
369+
getNodeColors(0, 100, false, 'errorRate', false).borderWidth,
370+
);
371+
});
372+
373+
it('does not treat other metrics as no-data', () => {
374+
expect(getNodeColors(10, 20, false, 'throughput', false)).toEqual(
375+
getNodeColors(10, 20, false, 'throughput'),
376+
);
377+
});
378+
379+
it('renders the border darker than the fill for the neutral state too', () => {
380+
const { backgroundColor, borderColor } = getNodeColors(
381+
0,
382+
100,
383+
false,
384+
'errorRate',
385+
);
289386
expect(parseHsl(borderColor).l).toBeLessThan(parseHsl(backgroundColor).l);
290387
expect(parseHsl(borderColor).h).toBe(parseHsl(backgroundColor).h);
291388
});
@@ -314,8 +411,8 @@ describe('getNodeColors', () => {
314411
});
315412

316413
it('returns different colors for different inputs', () => {
317-
expect(getNodeColors(5, 20, false)).not.toEqual(
318-
getNodeColors(10, 20, false),
414+
expect(getNodeColors(5, 20, false, 'throughput')).not.toEqual(
415+
getNodeColors(10, 20, false, 'throughput'),
319416
);
320417
});
321418
});
@@ -340,7 +437,7 @@ describe('getNodeColors', () => {
340437

341438
describe('getMetricGradientCss', () => {
342439
it('builds a left-to-right gradient from the ramp endpoints', () => {
343-
const css = getMetricGradientCss('errorRate');
440+
const css = getMetricGradientCss('latency');
344441
const stops = css.match(/hsl\([^)]+\)/g) ?? [];
345442
expect(css).toContain('linear-gradient(to right,');
346443
expect(stops).toHaveLength(2);
@@ -354,6 +451,27 @@ describe('getMetricGradientCss', () => {
354451
expect(parseHsl(stops[0]).h).toBe(SERVICE_MAP_METRIC_HUE.throughput);
355452
expect(parseHsl(stops[1]).h).toBe(SERVICE_MAP_METRIC_HUE.throughput);
356453
});
454+
455+
it('reuses the node fills verbatim as its error-rate stops', () => {
456+
const stops =
457+
getMetricGradientCss('errorRate').match(/hsl\([^)]+\)/g) ?? [];
458+
const fillAt = (errorPercentage: number) =>
459+
getNodeColors(errorPercentage, 100, false, 'errorRate').backgroundColor;
460+
expect(stops).toEqual([fillAt(0), fillAt(0.5), fillAt(3), fillAt(10)]);
461+
});
462+
463+
it('emits hard stops for error rate, one per bucket plus the neutral', () => {
464+
const css = getMetricGradientCss('errorRate');
465+
const stops = css.match(/hsl\([^)]+\)/g) ?? [];
466+
expect(stops).toHaveLength(4);
467+
// Hard stops, not a blend: each color carries an explicit start and end.
468+
expect(css).toContain('0% 25%');
469+
expect(css).toContain('75% 100%');
470+
expect(parseHsl(stops[0]).h).not.toBe(SERVICE_MAP_METRIC_HUE.errorRate);
471+
for (const stop of stops.slice(1)) {
472+
expect(parseHsl(stop).h).toBe(SERVICE_MAP_METRIC_HUE.errorRate);
473+
}
474+
});
357475
});
358476

359477
describe('getServiceMetricValue', () => {

0 commit comments

Comments
 (0)