Conversation
Drives a non-production realm server with the load a cohort of users produces: many sessions holding unbounded dashboard queries open while a few sessions write cards into the same realm. That shape is write-driven — each write invalidates the index, the realm broadcasts it, and every connected client re-runs its live searches — so a read-only replay reproduces none of it. The harness lives in scripts/ beside bench-realm, runs as .ts through Node's type stripping like its siblings, and imports nothing from the workspace: it has to run from inside the realm server's own AWS region, where there is no checkout, so installing it is a copy rather than a build. A driver on the far side of a home connection measures its own link, which is why every search reports headers and body separately and the summary warns when transfer dominates. Which queries the readers issue and what the writers write is a workload file rather than code, written in the `_federated-search` entry wire grammar; workload.example.json is the shape. The scripts refuse to run against any boxel.ai host, with no override flag. tests/load-harness-test.ts covers the parts a run cannot: the guard, credential parsing, argument parsing, workload validation, and the realm-event skip test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
`handleRequestForward` looks the destination up in AllowedProxyDestinations and rejects an unlisted one before it takes `withUserCostLock`. So a call aimed at a refused destination never reaches the lock, and cannot stand in for the serialization between one user's successive generations. State what the flag does reach — JWT verification, body parsing, and the `proxy_endpoints` lookup — and name the boundary, so a run is not read as evidence about lock contention. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 092cf5ceba
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| Authorization: session.serverToken, | ||
| }, | ||
| body: JSON.stringify({ | ||
| url: 'https://load-harness.invalid/v1/chat/completions', |
There was a problem hiding this comment.
With --model-calls, this .invalid URL is rejected by AllowedProxyDestinations.getDestinationConfig in packages/realm-server/handlers/handle-request-forward.ts:179-191, before the handler reaches withUserCostLock at line 293. Consequently the recorded 400s measure only allowlist rejection and never acquire or contend on the user cost lock, so this mode cannot support its stated conclusion; use a configured no-cost destination or another path that reaches the lock.
Useful? React with 👍 / 👎.
| **One manual step remains.** The other users need read access to the realm. | ||
| Granting it is not a `boxel realm` subcommand, so it is a UI action or a direct | ||
| API call. Each session authenticates as its own user: searches authorize per | ||
| realm, and the per-user cost lock that serializes model calls is keyed by Matrix | ||
| user, so a single shared account reproduces neither. |
There was a problem hiding this comment.
Following these setup instructions with the default --writers 2, only the first credential owns the realm while every other user is granted read access. run-load.ts then assigns the first two authenticated sessions as writers, so the second writer's POSTs receive 403 responses and the default run produces roughly half its intended write and invalidation rate; the setup must grant write access to the first N writer accounts or distinguish writer credentials explicitly.
Useful? React with 👍 / 👎.
| function announceInvalidation() { | ||
| let waiters = invalidationWaiters; | ||
| invalidationWaiters = []; |
There was a problem hiding this comment.
In the default mode, announceInvalidation clears and resolves only readers currently blocked in nextInvalidation; if a write completes during the initial Promise.all or while a reader is executing a previous search set, there is no registered waiter and the invalidation is discarded. Once searches take longer than the write interval—the saturation scenario this harness targets—the driver therefore underproduces the browser's follow-up searches; retain a pending generation/count per reader so an in-flight reader runs again after it finishes.
Useful? React with 👍 / 👎.
| stats.eventsSkipped += batch.length * (specs.length - due.length); | ||
| stats.eventsMatched += batch.length * due.length; |
There was a problem hiding this comment.
When multiple realm events accumulate, each due query is executed only once at line 429, but these counters add batch.length * due.length and the summary later labels that value as reruns "performed." A query matching one event in a five-event batch is therefore reported as five reruns even though only one request was made, corrupting the primary --subscribe result; count the actual due.length executions separately from event/query match decisions.
Useful? React with 👍 / 👎.
| `stamped with an x-boxel-logging-correlation-id, so a slow one here can be\n` + | ||
| `joined to the server's own timing by its corr= id.`, | ||
| ); | ||
| process.exit(0); |
There was a problem hiding this comment.
When the duration expires or Ctrl-C invokes finish, this immediate exit terminates any searches, writes, or model calls still in flight after printing the summary. Under saturation those outstanding operations are disproportionately the slowest ones, so omitting them biases tail latency and error counts downward; track the loop promises and await or explicitly abort and account for them before exiting.
Useful? React with 👍 / 👎.
| let minutes = args.minutes || 1; | ||
| console.log(`\nrate: ${Math.round(stats.searches / minutes)} searches/min`); |
There was a problem hiding this comment.
The README explicitly supports ending a run early with Ctrl-C, but this denominator remains the configured duration. For example, stopping a ten-minute run after one minute reports one tenth of the actual searches per minute, which can invalidate comparisons between runs; record the start time and divide by the elapsed duration instead.
Useful? React with 👍 / 👎.
| } | ||
| return; | ||
| } | ||
| record(spec.label, headersMs, bodyMs, text.length); |
There was a problem hiding this comment.
text.length counts UTF-16 code units in the decoded response, not bytes transferred, so card data containing non-ASCII text yields incorrect per-query KB and throughput figures; transparent content decoding can widen the discrepancy further from the claimed network-byte measurement. Read/count the response as bytes, or clearly report an uncompressed encoded payload size instead.
Useful? React with 👍 / 👎.
habdelra
left a comment
habdelra
left a comment
There was a problem hiding this comment.
[Claude Code 🤖] This review went after one question: does each number the harness prints mean what the README, the skill and the inline comments say it means — plus the auth/permission path a run depends on. The prose here is load-bearing (the whole point is that a number quoted wrong becomes a wrong platform conclusion), so I traced every such claim to the code it rests on: handleRequestForward, search-entries.ts's subscription callback and #indexEventCannotMatch, wireFilterTypeAnchors, and fetch's own encoding behaviour. I did not run the harness against a realm server, so nothing here speaks to the reference numbers in the skill.
One blocking defect in the default configuration (the second writer is a read-only user, so the default run writes at half its stated rate) and one in the --subscribe headline count (re-runs are counted per event × query, but issued once per batch). The rest is accuracy of the claims around otherwise sound machinery.
On the bot's --model-calls thread: no action needed. The head commit already states exactly that bound — in modelCall's comment, the summary's own printed note, the README and the skill — and the bot is reviewing the commit before it. I confirmed the ordering it describes (getDestinationConfig rejects at handle-request-forward.ts well ahead of withUserCostLock), which is what the code now says.
Recommendations, in the order I'd take them:
- Make the setup grant write to the first
--writersaccounts, or default--writersto 1 — see the thread on the README's manual step. - Count re-runs actually issued for the
--subscribeline — see the thread on theeventsMatched/eventsSkippedaccumulation. - Extend the skip-test caveat (or the skip test) to cover
prerender_html, full-index events and nestedany/everyanchors; it currently errs in both directions while the caveat names only one — see the thread oneventCannotMatch. - Measure bytes as bytes, and decide what "transferred" means under gzip — see the thread on
record(..., text.length). - Don't drop an invalidation announced mid-search-set — see the thread on
announceInvalidation. - Base the rate on elapsed time, and account for in-flight requests at exit — see the thread on the rate line.
- Reject
NaNfrom a numeric flag — see the thread onparseArgs. - Non-blocking, structural: the two caveats that decide how a result may be quoted (the
--subscribedivergence, the--model-callsboundary) are now written out in four places each —run-load.ts's header,lib/realm-events.ts, the README and the skill. They're the paragraphs most likely to go stale and the most expensive to have stale. Consider one canonical home (the code comment) with the README and skill pointing at it, so a future fix to the skip test doesn't leave three copies claiming the old behaviour.
Adjacent, out of scope: writerLoop seeds n = index * 1000, so two writers collide after 1000 writes each — unreachable at the documented cadences, worth a note if the interval ever drops.
Generated by Claude Code
| **One manual step remains.** The other users need read access to the realm. | ||
| Granting it is not a `boxel realm` subcommand, so it is a UI action or a direct | ||
| API call. Each session authenticates as its own user: searches authorize per | ||
| realm and realm events are broadcast into each user's own session room, so a | ||
| single shared account reproduces neither. |
There was a problem hiding this comment.
[Claude Code 🤖] Following this step as written leaves the default run writing at half its configured rate, and the invalidation rate is the load. setup-realm.ts makes creds[0] the owner and this paragraph grants everyone else read; run-load.ts then takes the writers off the front of the session list (sessions.slice(0, writers)), so with the default --writers 2 the second writer is a read-only user and every one of its POSTs comes back 403. The 403s land in writeErrors, which is printed — but a run still produces a summary that looks complete.
Either grant write to the first --writers accounts here (and say so in the skill's step 2, which carries the same "grant the other users read access" wording), or default --writers to 1 so the documented setup and the default flags agree.
Regression, blocking — it's the default configuration. This confirms the bot's read on the same lines; I traced the assignment to writerSessions in run-load.ts and the owner selection in setup-realm.ts.
Generated by Claude Code
| stats.eventsSkipped += batch.length * (specs.length - due.length); | ||
| stats.eventsMatched += batch.length * due.length; |
There was a problem hiding this comment.
[Claude Code 🤖] These two counters are (event × query) decision pairs, but the summary prints eventsMatched as query re-runs: N performed — and the line below it executes each due spec exactly once per batch, not once per event. A batch of five events where one query is due reports 5 re-runs against 1 search actually issued, so the headline --subscribe number overstates by the batch factor, which grows precisely as the run saturates and batches get bigger.
Count the executions separately from the decisions:
stats.reRunsPerformed += due.length;
stats.pairsSkipped += batch.length * (specs.length - due.length);
stats.pairsMatched += batch.length * due.length;and print reRunsPerformed on the "re-runs performed" line, keeping the pair counts for the skip percentage (which is a coherent ratio as it stands — just not a count of requests).
Regression, blocking: this is the number the mode exists to produce.
Generated by Claude Code
| // ONE DIFFERENCE, AND IT DECIDES WHAT A RESULT MEANS. The host resolves a | ||
| // query's type keys through its module loader, so a filter that names a type | ||
| // through a re-exporting module still matches rows stamped with that type's | ||
| // canonical defining-module spelling. The harness has no loader: it compares | ||
| // the literal `module/name` the workload file gave it. On a realm whose queries | ||
| // name their types directly the two agree. On a realm that filters through a | ||
| // re-export, the harness skips where a browser re-runs, and understates the | ||
| // traffic. Check which kind of realm you are pointed at before trusting a low | ||
| // re-run count. |
There was a problem hiding this comment.
[Claude Code 🤖] The caveat names one divergence and gives it a direction ("skips where a browser re-runs, and understates"). Reading search-entries.ts against this file, there are two more, and one of them runs the other way — so a reader who trusts this paragraph knows the sign of the error and has it wrong.
prerender_htmlevents. The host's subscription callback re-runs oneventName === 'prerender_html'as well as on incremental index events (search-entries.ts, theisIncrementalIndex/prerender_htmlguard), and the skip test is applied only to the incremental-index branch. Here every non-indexevent returnstrue, so those never produce a request — andload-harness-test.ts's "a non-index event never triggers a re-run" pins that as intended. Understates.- A full index event. The host takes the early return for
indexType !== 'incremental'(no re-run); hereeventName === 'index'with noinvalidatedTypesreturnsfalse, i.e. re-run. Overstates. - Nested type anchors.
wireFilterTypeAnchorswalksevery/anybranches;typeKeysOfinworkload.tsreads only the top-levelfilter['item.on'], so a workload query anchored inside aneverygetsundefinedkeys and re-runs unconditionally where the host would skip. Overstates.
Cheapest fix for the first two is in this function (gate on indexType, and treat prerender_html as a re-run trigger); the third could be a wireFilterTypeAnchors-shaped walk in typeKeysOf. If any stays as-is, extend this comment, the README's second bullet and the skill's caveat to say the count can err in both directions and name which events do it.
Regression (the claim is new with this PR), non-blocking if the caveat is corrected — the counts are already labelled indicative.
Generated by Claude Code
| } | ||
| return; | ||
| } | ||
| record(spec.label, headersMs, bodyMs, text.length); |
There was a problem hiding this comment.
[Claude Code 🤖] text.length is UTF-16 code units of the decoded body, which is the one number the README and skill both stake their reputation on ("Byte counts are trustworthy regardless of where the driver runs"). Two gaps, both measured against a local server just now rather than argued:
- Node's
fetchsendsaccept-encoding: gzip, deflateby default and decompresses transparently. On a gzip'd response my probe readtext.length208 where the wire carried 42 bytes. - Non-ASCII inflates the other way: the same body was 458 UTF-8 bytes against those 208 code units. Card titles with an em dash or an emoji are enough.
Buffer.byteLength(text) fixes the second and makes the number an honest decoded payload size, which is what the per-shape KB column is for. For the first, either read response.headers.get('content-encoding') and say in the summary which one is being reported, or send Accept-Encoding: identity so decoded and transferred coincide — worth deciding explicitly, since MB/s is currently decoded bytes over transfer seconds and will read high wherever a hop compresses.
Regression, non-blocking, but it undercuts the claim the tool is built around.
Generated by Claude Code
| function announceInvalidation() { | ||
| let waiters = invalidationWaiters; | ||
| invalidationWaiters = []; | ||
| for (let resolve of waiters) { | ||
| resolve(); | ||
| } | ||
| } | ||
| function nextInvalidation(timeoutMs: number): Promise<void> { | ||
| return new Promise<void>((resolve) => { | ||
| invalidationWaiters.push(resolve); | ||
| setTimeout(resolve, timeoutMs); | ||
| }); | ||
| } |
There was a problem hiding this comment.
[Claude Code 🤖] An invalidation announced while a reader is inside its Promise.all has no registered waiter and is dropped — the reader then sits in nextInvalidation until the next write or the 60 s idle floor. The loss rate is proportional to how long a query set takes relative to --write-every-ms, so the default mode underproduces load exactly in the saturation regime the harness is for, and quietly: nothing counts a dropped wake-up.
The --subscribe path already gets this right with its queue + wake. The same shape here is a per-reader pending flag:
let pending = false; // per reader
// announce: pending = true for each reader, then resolve any waiter
// loop: if (pending) { pending = false; } else { await nextInvalidation(idleReRunMs); }Regression, non-blocking — the run still produces load, just less of it than --write-every-ms implies. Worth a counter either way so a run can report how many wake-ups it coalesced.
Generated by Claude Code
| let minutes = args.minutes || 1; | ||
| console.log(`\nrate: ${Math.round(stats.searches / minutes)} searches/min`); |
There was a problem hiding this comment.
[Claude Code 🤖] Two things bias the summary when a run doesn't end exactly on schedule, and the README advertises both endings.
The rate divides by the configured --minutes, so a ten-minute run stopped after one with Ctrl-C reports a tenth of the real searches/min — and the skill quotes "889 searches/min sustained" as a comparable figure across runs. Stamp a start time and divide by the elapsed wall clock.
And finish() ends with process.exit(0), killing whatever searches and writes are in flight. Under saturation those are the slowest requests by selection, so p90/p99 and the error counts are biased low precisely where the tail is the thing being measured. Tracking the loop promises and awaiting them with a bounded grace period (or counting the abandoned ones in the summary) would keep the tail honest.
Regression, non-blocking.
Generated by Claude Code
| out[camel(key)] = | ||
| typeof spec[camel(key)] === 'number' ? Number(value) : value; | ||
| } |
There was a problem hiding this comment.
[Claude Code 🤖] A mistyped numeric flag becomes NaN and the run reports it as a success. --minutes lots (or --minutes with the value swallowed by a preceding valueless flag) gives setTimeout(finish, NaN), which fires on the next tick: the harness authenticates, prints the summary with searches: 0, and exits 0 — and the rate line reads normally because NaN || 1 is 1. --readers/--writers as NaN go the same way through slice.
load-harness-test.ts pins the NaN as the behaviour ("a number flag given a non-number yields NaN"); a run is expensive to set up and this failure mode looks like a completed run, so it's worth rejecting instead:
let n = Number(value);
if (Number.isNaN(n)) {
throw new Error(`--${key} needs a number, got: ${String(value)}`);
}Also worth refusing an unknown --flag here: one that isn't in the spec and isn't boolean silently consumes the next argv entry as its value, so a typo can eat a real flag's argument.
Regression, non-blocking.
Generated by Claude Code
Two ways to run comparable load tests without passing a config file
around.
workload.experiments.json targets the experiments realm, which ships in
the repo and exists in every deployed environment, so it is the one realm
everybody can point at. Its types are that realm's highest-count instance
types, picked by count rather than by interest. The repo realm and a
deployed one are not kept in step — Spec is 117 instances deployed
against 28 in the repo, CardListing 26 against 0 — so the file records
both columns and says what that buys: comparability across runs against
one target, not across targets.
--derive-workload builds the queries from the realm under test instead,
ranking its own /_types summary by instance count and querying the top
--derive-top. Two people testing one realm then need no shared config,
and --emit-workload turns the result into a committable file whose
realm-local modules are written back as ${realm} so it travels between
clones. A 401 or 404 from /_types stops the run rather than falling back
to a built-in workload: two people believing they ran the same test, and
not having done, is worse than a failed run.
The per-shape summary now states the two things its table contained but
did not say — what one pass over every shape transfers, and how far
server time spreads across shapes when the spread is wide enough that an
aggregate describes none of them.
Readers outnumber writers when sessions are short, so --writers 0 is a
read-only run against a realm nobody wants dirtied.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two ways the harness could report a number for a run nobody asked for. A derived workload picked the highest-count type, and fell back to one defined outside the realm when no realm-local type ranked — which the comment directly above it said was not a valid target, because such a module is not addressable relative to the realm. On a deployment's shared realms every top type is external, so the common case emitted a write block that fails at the first POST. The write target is now realm-local or absent: a workload may carry no write block, the derivation says on stderr which types ranked and why none qualified, and run-load refuses to start writers without one rather than inventing a target. Those realms are granted read-only anyway, so --writers 0 is the run that works there, and the README and skill say so where they recommend a target. A flag the spec does not define was written to the parsed args and never read, so `--duration 45` ran for the default ten minutes with nothing on stderr. Unknown options are now rejected, listing every one of them and the valid spellings — which also protects `--writer 0`, a typo that otherwise leaves two writers against a realm you may only be able to read. The standard workload now says roughly what one pass transfers on a deployed realm, so the cost of a long run from CloudShell is known before it starts rather than after. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`fetch` resolves when response headers arrive, so a request that had to open a socket first has a TCP and a TLS handshake inside the promise — and `headers` reported all of it as if the server had been thinking. Readers re-run on an interval far longer than undici's keep-alive, so on a realm nobody is writing to, which --writers 0 makes the normal case, nearly every sample paid it. Against a local server, 59 of 60 timed searches opened a connection. The driver now opens a batch's connections before starting the clock, and 0 of 60 timed searches open one. Priming matches the batch's concurrency: N concurrent requests want N sockets, and undici hands a request to an already-free client in preference to opening another, so a single primer would funnel the batch onto one socket and change the concurrency under measurement. A socket also returns to the pool a tick after its response settles, so the primer yields before the batch dispatches — without that, the last search still opens its own connection. The primer is a CORS preflight, answered ahead of the router, so it costs the server nothing beyond the connection it exists to open. What a cold socket costs on the current link is measured at startup and printed, so the correction is visible either way, and --prime-connections=false relabels `headers` as including setup rather than quietly containing it. The docs no longer call `headers` the platform number: it is server work plus a round trip, comparable between runs from one place. Turning an on-by-default option off needs `--flag=false`, so boolean options now take an explicit value instead of reading 'false' as truthy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The reported setup cost subtracted the second request from the first. The first is cold, but so is the second: a socket returns to undici's pool a tick after its response settles, so a request issued immediately after opens its own connection. The subtraction compared two handshakes against each other and reported the jitter between them as the cost of a handshake — against a control server charging a known 295 ms, it read 11 ms, and a rerun read 48 ms. With different jitter it can land near zero, and a clamp at zero hid the negative case entirely. The warm sample now comes after a yield, and is the cheapest of several rather than a single one, so a slow warm sample cannot shrink the figure — under-reporting is the damaging direction, since the number exists to stop someone reading setup as server time. The same control now reads 295 and 296 ms. Both raw samples are printed beside the difference, so a wrong subtraction is visible rather than authoritative, and the figure is no longer clamped: in-region it belongs in the noise and says so. The pool property both the primer and the measurement depend on now lives in one module with them. Written apart, one yielded and the other did not. Also states, where the harness claims what `headers` means, that it rises with --readers: that is the server under concurrent load, which is the thing being measured rather than an artifact to correct for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`_federated-search` serves two documents from one filter, selected by the
`fields[entry]` sparse fieldset: the prerendered renderings a grid
displays, or the card serializations a query-backed field instantiates
from. `store.search` sends `fields: { entry: ['item'] }`, so the second is
the query-backed-field path — and on a deployed realm the first costs
four to six times as much for the same filter, 94 KB against 405 and 537.
The driver sent no fieldset, so every number it produced described a
grid, whatever the workload's queries said. `--fieldset` and a workload
`fieldset` member now select the path, an explicit flag beating a
committed file; `item-html` is its own value rather than a combination,
since the wire accepts that pair and nothing else alongside `item`. The
default stays `entries` so existing numbers remain comparable, and both
committed workloads pin a value rather than leaving it implicit. Every
run states the path it modelled in its header and its summary: a payload
or latency figure quoted without its path is not interpretable.
A workload's filter was already passed through whole, so `eq` /
`contains` / `range` / `any` ride alongside the `item.on` anchor — which
is what a run needs to A/B adding a predicate to a query that has none.
That is now pinned by a test and shown by committed examples.
A derived workload is the unmitigated shape by construction: the type
summary carries names and counts, so every derived query is type-only
with no predicate. Faithful as a reproduction, useless as a measurement
of a mitigation that adds one, and the docs say so where the flag is
described.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3 participants
A load harness for the realm server, under
packages/realm-server/scripts/load-harness/, plus a Claude skill covering how to run it so the numbers mean something.What it does
Authenticates N real Matrix users against a non-production realm server, then holds
_federated-searchqueries open from--readerssessions while--writerssessions POST cards into the same realm. It reports per-query-shape payload bytes, headers-vs-body latency split, write latency, and — under--subscribe— realm-event re-run counts.The load it reproduces is write-driven, which is the part most easily got wrong. Each write invalidates the index, the realm broadcasts it, and every connected client re-runs its live searches, so a read-only replay reproduces none of the interesting behaviour.
Which path a run measures
_federated-searchserves two documents from the same filter, selected by thefields[entry]sparse fieldset, and they are not close in cost:--fieldsetentries(default)html+css)itemfields: { entry: ['item'] }item-htmlfields: { entry: ['item','html'] }store.searchsendsfields: { entry: ['item'] }(packages/host/app/services/store.ts) and instantiates cards live from the result, soitemis the path a query-backed field takes — and the path a search-saturation investigation is usually about. A driver that sends no fieldset measures what a grid costs, whatever its workload's queries say.A workload file pins the choice with a
"fieldset"member and--fieldsetoverrides it; both committed workloads pin a value rather than leaving it implicit.item-htmlis its own value rather than a flag combination, since that is the only other pairing the wire accepts. The default staysentriesso existing numbers remain comparable, and every run prints the path it modelled in its header and its summary — a payload or latency figure quoted without its path is not interpretable.A workload's filter is passed through whole, so
eq/contains/range/anyride alongside theitem.onanchor. That is what a run needs to A/B "add a predicate to a query that has none";workload.example.jsoncarries committed examples and a test pins the pass-through.--derive-workloadproduces the unmitigated shape by construction — the type summary carries names and counts, so every derived query is type-only with no predicate. Faithful as a reproduction, useless as a measurement of a mitigation that adds one. The docs say so where the flag is described.Where the queries come from
Three modes, in increasing order of specificity. Nobody has to pass a config file around to get comparable numbers.
workload.experiments.json— the standard workload, and the recommended start. It targets the experiments realm, which ships in the repo aspackages/experiments-realmand exists in every deployed environment, so it is the one realm everybody can point at. Its eight shapes are that realm's highest-count instance types, picked by count rather than by interest.It is comparable across runs against the same target, not across targets, and the file says so with both columns side by side — the repo realm and a deployed one are not kept in step:
SpecFileDefFormatPreviewAuthorCardListing--derive-workload— for the realm you actually care about. It reads the realm's ownGET <realm>/_typessummary, keeps thekind: 'instance'entries, ranks byattributes.total(id as tiebreak, so a tie cannot reorder the selection between runs), splits eachidat the last/into anitem.onanchor, and queries the top--derive-top(default 8) at--derive-page-size(default 20;0is unbounded). Both id spellings in circulation split correctly — the prefix form@cardstack/base/spec/Specand the URL formhttps://…/experiments/author/Author.--emit-workload <path>(-for stdout) writes the derived workload and exits, which is how a derived workload becomes a shared one. Realm-local modules are emitted as${realm}…, so the file travels between clones rather than pinning to the realm it came from._typesneeds the realm's own JWT — a bare session token gets a 401 — and a failure there stops the run rather than falling back to a built-in workload: two people believing they ran the same test and not having done is worse than a failed run.A derived workload can legitimately come back read-only. The write target has to be a type defined in the realm itself; an external module is not addressable relative to the realm, so a POST naming one would be a guess discovered at write time. When no selected type qualifies the
writeblock is omitted, stderr names the types that did rank and says why, and the run needs--writers 0. That is the expected outcome on a deployment's shared realms, where every top type is@cardstack/…— and it matches those realms being granted read-only, which makes--writers 0the only run that works there rather than a convenience. The README and skill say so where they recommend a target.A hand-written file for reproducing a specific card's query pattern.
workload.example.jsonis the shape.All three are written in the
_federated-searchentry wire grammar: the type anchor isitem.on, field paths carry anitem.prefix.Other notable decisions
Where it lives and how it runs.
scripts/besidebench-realm/, the existing precedent for a multi-file operational tool. The files are.tsrun directly through Node's type stripping, like every sibling script;pnpm load-harnessandpnpm load-harness:setupare the package entry points.It imports nothing from the workspace. The driver has to run from inside the realm server's own AWS region, on a box with no checkout and no
node_modules, so installing it is a copy rather than a build. Globalfetchandnode:built-ins only. The repo's existing erasable-syntax lint rule is what keeps that copy runnable.Headers and body are timed separately, and connection setup is kept out of both.
fetchresolves when response headers arrive; reading the body is a second wait. Splitting them separates the server's own work from bytes crossing the network — a driver on the far side of a home connection measures mostly its own link, and the summary warns when the body leg dominates.But
fetchalso resolves only after any TCP and TLS handshake the request had to perform, soheaderswould silently contain connection setup. Readers re-run on an interval far longer than undici's keep-alive, so on a realm nobody is writing to — what--writers 0makes normal — nearly every sample pays it: measured against a local server, 59 of 60 timed searches opened a connection. The driver now opens each batch's connections before starting the clock, and that becomes 0 of 60, with no increase in total connections — the setup just moves outside the timed window.Priming matches the batch's concurrency, because N concurrent requests want N sockets and undici hands a request to an already-free client in preference to opening another; a single primer would funnel the batch onto one socket and change the concurrency being measured. A socket also returns to the pool a tick after its response settles, so the primer yields before the batch dispatches — without that the last search still opens its own connection. The primer is a CORS preflight, answered by
@koa/corsahead of the router, so it costs the server nothing beyond the connection it exists to open.What a cold socket costs on the current link is measured at startup and printed, and
--prime-connections=falserelabelsheadersas including setup rather than quietly containing it.That measurement carries the same trap as the primer, so it lives in the same module: the first request after process start is cold, and so is the second, because the socket has not returned to the pool yet. A cold-minus-next subtraction therefore compares two handshakes and reports the jitter between them — against a control server charging a known 295 ms handshake it reads 11 ms on one run and 48 ms on another. The warm figure is now the cheapest of several samples taken after a yield, which reads 295–296 ms against the same control; both raw samples are printed beside the difference so a wrong subtraction is visible, and the figure is not clamped at zero, since in-region it belongs in the noise and should say so. The docs no longer call
headers"the platform number": it is server work plus a round trip, comparable between runs from one place and from a fixed--readerscount, sinceheadersrises with reader count — that is the server under concurrent load, which is the thing being measured rather than an artifact to correct for. Byte counts remain trustworthy wherever the driver runs; latency is not.The summary states two things its per-shape table contained but did not say: what one pass over every shape transfers (that is what opening a screen costs, not any single row), and how far server time spreads across shapes when the spread is wide enough that an aggregate describes none of them. On a deployed experiments realm that spread is 6×, and it does not track instance count — the 117-instance type is the fastest of the six.
--model-callsis bounded by where the handler rejects. It puts a_request-forwardcall before each write, aimed at a destination the realm server refuses, so it spends no tokens. Worth knowing precisely how far that gets:handleRequestForwardverifies the JWT, parses the body, and looks the destination up inAllowedProxyDestinations— and rejects there, in front ofwithUserCostLock. So the flag adds a second authenticated round trip per write but does not reproduce per-user cost-lock contention; reaching the lock needs an allowlisted destination and therefore real spend. The comments, README and skill all say so, because a run read as evidence about that lock would be wrong.Two invalidation modes, and the difference matters when quoting a result. By default readers re-run on this driver's own writes, which models the fan-out.
--subscribereads the realm'sapp.boxel.realm-eventstream and applies the host's skip test, which measures it — but the harness compares type keys literally where the host resolves them through its module loader, so its re-run counts are indicative rather than authoritative. Payload and latency are faithful in both modes.Boolean options take an explicit value. An option that defaults to on can only be disabled by
--flag=false, and reading that string as truthy would leave it on while the operator believed otherwise.Unknown options are rejected. A flag the spec does not define used to be written to the parsed args and never read, so a mistyped
--duration 45ran for the default ten minutes with nothing on stderr. It now fails, listing every unrecognised option and the valid spellings — which also catches--writer 0, a typo that would otherwise leave two writers against a realm you may only be able to read.The production guard has no override. Any
boxel.aihost is refused. This drives a realm server to saturation on purpose; an operator under time pressure will reach for a--forcethat exists.No credential file or realm-specific workload can be committed from that directory. Its
.gitignorerefuses*.csvand anyworkload*.jsonother than the two shared ones.Tests
packages/realm-server/tests/load-harness-test.tscovers what a run cannot: the production guard, credential parsing, argument parsing (including unknown-option rejection), workload validation and${realm}/${n}substitution, the_types→ workload derivation (ranking, file-kind exclusion, both id spellings, page sizing, tie stability, label collisions, write-target selection, and the read-only case where nothing realm-local ranks), the realm-event skip test, and the summary statistics. It also parses both committed workload files, so neither can drift out of the format and both are checked to pin a fieldset; the fieldset wire members, name validation and descriptions are covered; an arbitrary wire filter is asserted to survive the workload loader untouched; and the connection-setup sampler is driven through a replayed sequence of request durations so the cold-second-sample trap is pinned by a test rather than by prose. 58 tests, no database, no network.The fieldset resolution is additionally verified on the wire: the real driver is run against a server that records search request bodies, confirming that the default sends no
fieldsmember, a workload'sfieldsetis honoured, an explicit--fieldsetbeats the file, and a predicate survives alongside the type anchor.The connection-priming behaviour is verified separately by running the real driver against a local server that records which TCP socket each request arrived on — that is where the 59/60 and 0/60 figures come from.
🤖 Generated with Claude Code