Skip to content

evidence: live / replay / synthetic latency basis fixture - #22

Merged
roy-tong merged 1 commit into
roy-tong:mainfrom
modelpath-dev:evidence/live-replay-synthetic-latency-basis
Sep 21, 2026
Merged

roy-tong merged 1 commit into
roy-tong:mainfrom
modelpath-dev:evidence/live-replay-synthetic-latency-basis

Conversation

@modelpath-dev

Copy link
Copy Markdown
Contributor

Hi Roy,

Following up on our thread about counting grain and provenance. Here's the fixture I mentioned, laid out the same way as David's inference-boundary entry so it sits cleanly in the evidence directory.

The setup: one operation_id, three rows, each tagged with its provenance.

Row provenance latency_ms
row-live-1 live 100
row-replay-1 replay 20
row-synthetic-1 synthetic 300

The number pair you asked for:

  • live-only mean: 100 ms (correct production aggregate)
  • all-rows mean: 140 ms, i.e. (100 + 20 + 300) / 3 (the mixed-basis failure)

Replay gets its own view (20 ms) and synthetic stays out of production aggregates entirely. The point is that 140 looks like a perfectly reasonable latency number, which is exactly why it's dangerous.

Files:

  • conformance/evidence/live-replay-synthetic-latency-basis/README.md
  • conformance/evidence/live-replay-synthetic-latency-basis/fixture.json

It's a synthetic fixture only, no real provider traffic behind it. I didn't wire it into a runner, same as the inference-boundary entry. I did recompute the three means straight from the rows to make sure the expected values in the JSON line up.

Let me know if you'd like anything renamed or reshaped.

Chandan

@roy-tong
roy-tong merged commit bc6cdaa into roy-tong:main Sep 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants