fix(demos): flapping herd outage for faithful spikes-with-gaps

The herd's sustained outage produced a smooth rising ramp for
independent, un-synchronized naive clients, not the spikes-with-gaps
the tour copy claims. Switch the model input to a flapping backend
(down/recover x3) so unbounded retries genuinely surge on each dip
and clear on recovery -- real spikes-with-gaps, faithfully sourced
from backend recovery rather than client synchronization. The
existing simulateHerd algorithm already produces this shape once the
fault flaps; this is a scenario + rendering + copy change.

Add computeOutageBands so the rate strip shades each dip separately,
leaving recovery gaps visibly unshaded between them (falls back to
the single outage window for callers that don't pass bands, so the
full-stack macro strip is unaffected). Re-measure the naive
multiplier against the flapping scenario (~18x, down from the
sustained model's ~70x since each dip is shorter) and reconcile
retry.md's copy and the change-file's motivation/design/payoff
language to match.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: lesnik512

docs/demos/engine.js

@@ -184,6 +184,24 @@ window.HttpwareDemo = (function () {
     return { from, to: Math.min(scenario.dur, to + scenario.dur / steps), label };
   }
 
+  // All contiguous "backend in trouble" intervals (down or slow), for a strip that
+  // shades each outage window separately — so a flapping backend's recovery GAPS stay
+  // visibly unshaded between the shaded dips. Decorative; never drives the sim.
+  function computeOutageBands(scenario) {
+    const steps = 400, worst = () => 0;
+    const bands = [];
+    let start = null;
+    for (let i = 0; i <= steps; i++) {
+      const t = (scenario.dur * i) / steps;
+      const f = scenario.fault(t, worst);
+      const down = !f.ok || (f.ok && f.ms >= 1.0);
+      if (down && start === null) start = t;
+      else if (!down && start !== null) { bands.push({ from: start, to: t }); start = null; }
+    }
+    if (start !== null) bands.push({ from: start, to: scenario.dur });
+    return bands;
+  }
+
   // ---- shared guided-tour driver: spotlight cutout + side-placed coach ----
   // Extracted from mount() so a second mount mode (herd) reuses one implementation.
   // `els` supplies the tour DOM (dimT/dimB/dimL/dimR/ring/coach/cArrow/cStep/
@@ -254,15 +272,17 @@ window.HttpwareDemo = (function () {
   }
 
   // ---- thundering-herd simulation (faithful; reuses the real RetryBudget model) ----
-  // N independent clients hit ONE backend through a sustained outage. Every backend
-  // attempt (initial + each retry) is counted into a per-bucket series for each lane.
-  //   naive: FIXED backoff, NO jitter, UNBOUNDED retries -> same-bucket failures retry
-  //          into the same later bucket => synchronized spikes with dead gaps.
+  // N independent clients hit ONE backend through a FLAPPING outage (down, recover,
+  // down, recover...). Every backend attempt (initial + each retry) is counted into a
+  // per-bucket series for each lane.
+  //   naive: FIXED backoff, NO jitter, UNBOUNDED retries -> during each outage dip a
+  //          client retries until the backend recovers, so retries ACCUMULATE into a
+  //          surge that clears the instant the dip ends (its retry succeeds) -> a spike
+  //          per dip with a recovery gap between. The gaps are the backend healing, not
+  //          client synchronization (arrivals are phase-staggered, never lockstep).
   //   httpware: FULL-JITTER backoff + per-client RetryBudget + maxAttempts cap ->
   //          retry timing decorrelates into a near-constant rate; the budget and the
   //          attempt cap bound amplification. The budget is PER CLIENT, never shared.
-  // Arrivals are phase-staggered (not synchronized): the clustering that follows is
-  // caused by the retry policy, not by synchronized arrivals.
   function simulateHerd(scenario, opts) {
     const N = opts.clients, cfg = opts.retry, budgetCfg = opts.budget;
     const dur = scenario.dur, dt = HERD.dt, ri = HERD.reqInterval;
@@ -308,7 +328,7 @@ window.HttpwareDemo = (function () {
       return { series: arr, peak, total, mult: baseline > 0 ? peak / baseline : 0 };
     };
     return { naive: stat(naive), hw: stat(hw), buckets, dt, baseline,
-      outage: computeOutageWindow(scenario) };
+      outage: computeOutageWindow(scenario), outageBands: computeOutageBands(scenario) };
   }
 
   // Render a call-rate strip as an SVG of vertical bars into `el`. Buckets [0,
@@ -319,9 +339,10 @@ window.HttpwareDemo = (function () {
     const n = series.length, W = 100, H = 100, bw = W / n;
     const scale = opts.peakScale > 0 ? H / opts.peakScale : 0;
     let body = '';
-    if (opts.outage) {
-      const ox = (opts.outage.from / opts.dur) * W;
-      const ow = ((opts.outage.to - opts.outage.from) / opts.dur) * W;
+    const bands = opts.outageBands || (opts.outage ? [opts.outage] : []);
+    for (const b of bands) {
+      const ox = (b.from / opts.dur) * W;
+      const ow = ((b.to - b.from) / opts.dur) * W;
       body += `<rect x="${ox.toFixed(2)}" y="0" width="${ow.toFixed(2)}" height="${H}" class="herd-band"/>`;
     }
     const upto = Math.min(revealUpTo, n);
@@ -416,7 +437,7 @@ window.HttpwareDemo = (function () {
 
     function paint() {
       const peakScale = Math.max(sim.naive.peak, sim.hw.peak, 1);
-      const stripOpts = (cls) => ({ peakScale, baseline: sim.baseline, outage: sim.outage, dur: scenario.dur, cls });
+      const stripOpts = (cls) => ({ peakScale, baseline: sim.baseline, outage: sim.outage, outageBands: sim.outageBands, dur: scenario.dur, cls });
       renderRateStrip(els.naiveStrip, sim.naive.series, reveal, stripOpts('herd-bar-naive'));
       renderRateStrip(els.hwStrip, sim.hw.series, reveal, stripOpts('herd-bar-hw'));
       // running peak/total over revealed buckets so the numbers climb with the bars
@@ -430,7 +451,7 @@ window.HttpwareDemo = (function () {
       els.naiveTotalN.textContent = String(runTot(sim.naive.series));
       els.hwTotalN.textContent = String(runTot(sim.hw.series));
       // httpware is bounded by max_attempts, not a hard multiplier cap — it settles
-      // near ~3x for this scenario, never near naive's unbounded ~70x. Thresholds
+      // near ~3x for this scenario, never near naive's per-dip ~18x. Thresholds
       // reflect that measured gap, not the placeholder "<=2x is good" assumption.
       els.naiveMultN.style.color = nm > 5 ? 'var(--hw-bad)' : '';
       els.hwMultN.style.color = hm <= 4 ? 'var(--hw-ok)' : '';

docs/demos/retry.md

@@ -47,24 +47,27 @@ document.addEventListener('DOMContentLoaded', function () {
 
   HttpwareDemo.mountHerd('#retry-herd', {
     clients: 20,
+    intro: 'A real backend rarely dies cleanly — it <b>flaps</b>: fails, recovers, fails again. These strips show <b>backend call-rate over time</b> for twenty clients through three dips. Press play and watch the shape.',
     scenario: { id: 'storm', dur: 12.5,
-      fault: (now) => (now >= 2.0 && now < 9.0)
-        ? { ok: false, ms: 0.05, label: 'DOWN' } : { ok: true, ms: 0.05 } },
+      fault: (now) => {
+        const down = (now >= 2.0 && now < 4.0) || (now >= 5.5 && now < 7.5) || (now >= 9.0 && now < 11.0);
+        return down ? { ok: false, ms: 0.05, label: 'DOWN' } : { ok: true, ms: 0.05 };
+      } },
     retry: { maxAttempts: 3, baseDelay: 0.1, maxDelay: 5.0 },
     budget: { ttl: 10.0, minRetriesPerSec: 10.0, percentCanRetry: 0.2 },
     buildStops: (sim) => [
       { when: (s) => s.revealed >= Math.round(2.2 / sim.dt), spot: ['naiveStrip', 'hwStrip'],
-        title: 'The outage hits both herds',
-        body: 'Twenty naive clients and twenty httpware clients face the same sustained outage. Every failed request wants to retry. Watch what each herd does to the backend call-rate.' },
-      { when: (s) => s.revealed >= Math.round(4.0 / sim.dt), spot: ['naiveStrip'],
-        title: 'Naive: synchronized spikes',
-        body: 'All twenty naive clients back off the SAME fixed delay, so a batch that fails together retries together — the failures never spread out. Tall spikes with dead gaps between: the retry storm hammering a backend that is already down.' },
-      { when: (s) => s.revealed >= Math.round(6.5 / sim.dt), spot: ['hwStrip'],
-        title: 'httpware: a near-flat rate',
-        body: 'Full jitter scatters each client’s retries across the window instead of stacking them, and each client’s own budget + max_attempts cap how much it can add — so the aggregate holds at a low, steady few-times-baseline instead of exploding.' },
+        title: 'A flapping backend',
+        body: 'The backend drops, recovers, drops again — three dips (shaded). Every failed request wants to retry. Watch what each herd does to the backend call-rate through the dips.' },
+      { when: (s) => s.revealed >= Math.round(4.6 / sim.dt), spot: ['naiveStrip'],
+        title: 'Naive: a surge on every dip',
+        body: 'On each dip, twenty clients retry unbounded — the load piles up into a surge that keeps climbing until the backend recovers, then clears. Three dips, three spikes, with recovery gaps between: the retry storm hitting a backend every time it tries to come back.' },
+      { when: (s) => s.revealed >= Math.round(8.2 / sim.dt), spot: ['hwStrip'],
+        title: 'httpware: flat through every dip',
+        body: 'Full jitter spreads each client’s retries out, and each client’s own max_attempts=3 cap (with its per-client budget as the guarantee at higher volume) limits how much it can add — so httpware holds a low, steady few-times-baseline through every dip instead of spiking.' },
       { when: (s) => s.revealed >= sim.buckets - 1, spot: ['naiveMult', 'hwMult'],
         title: 'Peak load: the whole point',
-        body: 'The naive herd peaked at about 70× the healthy load — and keeps climbing the longer the outage runs, because its retries are unbounded. httpware peaked at about 3×: at this client count, that ceiling comes from each client’s own max_attempts=3 cap (jitter just spreads the retries flat instead of stacking them). The per-client budget is what holds the line as the herd grows even larger. Same outage, same client count — a flat rate also leaves the backend room to recover.' },
+        body: 'On its worst dip the naive herd spiked to about 18× the healthy load; httpware never exceeded about 3×, capped by each client’s max_attempts. That flat rate is exactly what lets the backend recover in the gaps — instead of being knocked back down by a retry surge every time it heals.' },
     ],
   });
 });

planning/changes/2026-07-19.02-resilience-demo-herd-and-enrich.md

@@ -1,5 +1,5 @@
 ---
-summary: Added a many-client thundering-herd macro view to the retry page (naive ~70x vs httpware ~3x peak backend load, httpware bounded by max_attempts at demo scale with a per-client budget) and folded three enrich techniques across the demo suite — a backoff countdown ring on timeout + circuit-breaker only (retry's sub-tick backoff makes a faithful ring there impossible, so it's dropped, not faked), a color-independent jagged failure shape everywhere, and a live backend call-rate macro strip with active-phase label on the full-stack page.
+summary: Added a many-client thundering-herd macro view to the retry page (naive ~18x vs httpware ~3x peak backend load through a flapping outage, httpware bounded by max_attempts at demo scale with a per-client budget) and folded three enrich techniques across the demo suite — a backoff countdown ring on timeout + circuit-breaker only (retry's sub-tick backoff makes a faithful ring there impossible, so it's dropped, not faked), a color-independent jagged failure shape everywhere, and a live backend call-rate macro strip with active-phase label on the full-stack page.
 ---
 
 # Design: Resilience demo herd view + enrich pass
@@ -9,7 +9,7 @@ summary: Added a many-client thundering-herd macro view to the retry page (naive
 Improve the resilience demo suite (shipped in `2026-07-18.02`) using techniques
 proven by the best existing visualizations. Two parts: (1) a **thundering-herd
 macro view** stacked below the single-client retry/budget demo — 20 naive clients
-vs 20 httpware clients, rate-into-the-backend over a sustained outage, so the
+vs 20 httpware clients, rate-into-the-backend over a flapping outage, so the
 retry storm is *shown* not just described; (2) an **enrich pass** folding three
 smaller techniques into the existing pages — a backoff/wait **countdown ring**, a
 color-independent **jagged failure shape**, and a small **macro strip** on the
@@ -24,11 +24,11 @@ post](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/)
 exposes one real gap and a few cheap wins:
 
 - **The thundering herd is invisible.** Our retry/budget demo is single-client, so
-  the one visual that makes "retry storm" and "why jitter" click — many clients
-  synchronizing into traffic spikes, jitter smoothing them into a flat rate — we
-  only assert in prose. AWS's canonical framing is a call-rate-over-time chart:
-  without jitter, "clusters of calls" with dead gaps; with jitter, an approximately
-  constant rate. We don't draw it.
+  the one visual that makes "retry storm" and "why jitter" click — a flapping backend
+  driving unbounded retries into spikes-with-gaps, jitter smoothing them into a flat
+  rate — we only assert in prose. AWS's canonical framing is a call-rate-over-time
+  chart: without jitter, "clusters of calls" with dead gaps; with jitter, an
+  approximately constant rate. We don't draw it.
 - **The backoff wait isn't shown.** A dot just eventually resolves; Encore renders a
   countdown timer *while a client waits between attempts*, making the delay tangible.
 - **Failures lean on color.** A failed request is a flat square distinguished mainly
@@ -41,17 +41,19 @@ single-client side-by-side, a distinct "Now scale it to 20 clients" section with
 own play button and mini-tour, preserving a micro → macro arc down the page (one
 client's retries, then the herd). A new engine **rate-strip render mode** (isolated
 from the existing lane renderer, selected by mount config) draws two stacked strips
-plotting calls-into-the-backend per tick over a *sustained* outage window:
-
-- **Naive lane:** 20 clients, fixed backoff, no jitter, unbounded retries → their
-  retries stay clustered into spikes separated by gaps (AWS's finding; we model
-  clustering, not literal lockstep — arrivals are spread, fixed backoff *preserves*
-  the clustering).
+plotting calls-into-the-backend per tick over a **flapping** outage — the backend
+goes down, recovers, goes down again, three dips with real recovery gaps between:
+
+- **Naive lane:** 20 clients, fixed backoff, no jitter, unbounded retries → on each
+  dip a client retries until the backend recovers, so retries accumulate into a
+  surge that clears the instant the dip ends — a spike per dip with a genuine
+  recovery gap between (AWS's finding; the gaps come from the backend healing, not
+  from client synchronization — arrivals are phase-staggered, never lockstep).
 - **httpware lane:** 20 clients, full-jitter backoff + per-client budget → jitter
-  decorrelates retry timing into a near-constant rate; at this client count each
-  client's `max_attempts=3` cap is what bounds its amplification (the per-client
-  budget's floor doesn't bind until far higher per-client volume — it's the guarantee
-  that holds as the herd grows).
+  decorrelates retry timing into a near-constant rate through every dip; at this
+  client count each client's `max_attempts=3` cap is what bounds its amplification
+  (the per-client budget's floor doesn't bind until far higher per-client volume —
+  it's the guarantee that holds as the herd grows).
 
 Both lanes reuse the *real* retry + budget models from the engine's source-of-truth
 constants block, one model instance per simulated client. **Fidelity framing that
@@ -63,15 +65,16 @@ imply a shared/global budget or that the budget (rather than `max_attempts`) doe
 capping at this scale.
 
 **Payoff metric.** The mini-tour spotlights the **peak load multiplier** — "backend
-saw N× baseline". Measured against the fixed seed: **naive ~70×** (unbounded retries
-explode and keep climbing with outage length) **vs httpware ~3×** (bounded by
-`max_attempts` at this client count) — the number reads directly off the tallest
-spike the strip already draws (spotlight number == visible fact). **Total calls**
-during the outage is a secondary, un-spotlighted counter (volume story). "The flat
-rate leaves the backend room to recover" is the qualitative closing beat, *not* a
-spotlighted stat — httpware's code models no backend-recovery dynamic, so no invented
-recovery number. ~4 beats: outage hits → naive's first clustered spike → httpware
-stays flat (jitter + the `max_attempts` cap) → peak-multiplier payoff.
+saw N× baseline". Measured against the fixed seed on the flapping scenario: **naive
+~18×** on its worst dip (unbounded retries surge each time the backend drops) **vs
+httpware ~3×** (bounded by `max_attempts` at this client count) — the number reads
+directly off the tallest spike the strip already draws (spotlight number == visible
+fact). **Total calls** during the outage is a secondary, un-spotlighted counter
+(volume story). "The flat rate leaves the backend room to recover" is the qualitative
+closing beat, grounded in the scenario itself (the backend genuinely recovers between
+dips) rather than an invented recovery number. ~4 beats: the flapping backend →
+naive's first dip-spike → httpware stays flat (jitter + the `max_attempts` cap) →
+peak-multiplier payoff.
 
 **Enrich pass.**
 
@@ -109,18 +112,21 @@ size. Jagged failure shape is one `demos.css` change.
 `node --check docs/demos/engine.js`. Scratchpad jsdom harnesses extended: the
 existing `verify-demos.js` / `verify-real-page.js` drive every stop including the new
 herd mini-tour to completion without stalling; a herd trace asserts naive peak
-multiplier ≫ httpware peak (dramatic gap, ~70× vs ~3×), httpware bounded near the
-`max_attempts` cap, and a fine bucket count for a smooth strip. Manual: retry page
+multiplier ≫ httpware peak (dramatic gap, ~18× vs ~3×) with the spikes-with-gaps
+shape itself (multiple separated naive spikes, one per flapping dip, with
+near-baseline recovery gaps between), httpware bounded near the `max_attempts` cap,
+and a fine bucket count for a smooth strip. Manual: retry page
 reads micro → macro top-to-bottom under mkdocs-material light and dark; countdown
 ring animates and empties on CB / timeout; failure shape distinguishable in
 grayscale; `prefers-reduced-motion` degrades to stepped updates.
 
 ## Risk
 
 - **Herd fidelity overclaim (medium × medium).** Easy to imply lockstep
-  synchronization or a shared budget. Mitigation: model clustering (not lockstep),
-  per-client budget instances, and copy reviewed specifically against the fidelity
-  framing above; reuse the real models rather than a bespoke herd approximation.
+  synchronization or a shared budget. Mitigation: spikes-with-gaps come from the
+  flapping backend's dip/recovery cycle (not lockstep), per-client budget instances,
+  and copy reviewed specifically against the fidelity framing above; reuse the real
+  models rather than a bespoke herd approximation.
 - **Rate-strip renderer as new surface (low × medium).** A second render mode adds
   engine branching and bug surface. Mitigation: isolate it from the lane renderer
   behind mount config; the full-stack macro strip reuses it, so one renderer serves