Skip to content

chore: measure gateway request rate during e2e runs - #2288

Open
nicosampler wants to merge 4 commits into
canton-network:mainfrom
BootNodeDev:measure-gateway-request-rate
Open

chore: measure gateway request rate during e2e runs#2288
nicosampler wants to merge 4 commits into
canton-network:mainfrom
BootNodeDev:measure-gateway-request-rate

Conversation

@nicosampler

@nicosampler nicosampler commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

What

Closes #1202.

Every e2e test now records the requests its browser sends to the wallet gateway. A script turns those recordings into a report. Nothing in the gateway changes.

Comments below are added for this description, the script does not print them.

RATE LIMIT REPORT   14 tests

limit evaluated                   200     # the limit being checked against
requests rejected (429)           0       # anything above 0 means the run was capped and cannot be used
busiest minute, one user          2366    # what a per user limit has to cover
requests before login             17      # these count against the per IP limit instead
users measured                    1

PER USER
user             busiest minute  top call   share
ledger-api-user  2366            ledgerApi  97%    # a single call is almost all of it

Three things to keep in mind about the numbers:

  1. Two runs of the same suite gave 1694 and 2366, so one run is a lower bound.
  2. This is LocalNet on a laptop, not production capacity.
  3. The limit is per user, so someone with several wallets spends one budget for all of them.

What we implemented

  1. A capture layer in core-wallet-test-utils that can be added to any Playwright test. It records the time, the sub of the bearer token, the JSON-RPC method and the response status. The token itself is never stored.
  2. Ping gets it through the shared fixture it already uses. Portfolio gets a version with the capture and nothing else, since its tests were not written against that fixture.
  3. A script that reads the captures and prints the report, plus a JSON file for comparing runs. It needs nothing running, so it also works on CI artifacts.

Alternatives we discarded

Gateway logs from CI. Truncated to the last ten thousand lines, which at debug level is twenty seconds to two minutes of a run. They also never carry the caller and the call on the same line, so traffic cannot be attributed to a user.

Playwright traces. They do contain everything we need. But CI only records them on a retry, so a passing run leaves nothing to read, and always recording them would grow artifacts that already reach tens of megabytes per job.

An access log in the gateway. The only source that counts exactly what the limiter counts, and useful beyond this ticket. It is production code and changes how the gateway behaves in operation, so it is your call rather than ours.

Notes

On the ticket's premise. The default went from 100 to 10000 in January. Back then portfolio had one test file and no polling, and was already going over 100. It now has seventeen tests and refetches every query every five seconds, and reaches 2366. Nobody touched the limiter in between. A number picked today will go stale the same way, which is why the reproducible measurement matters more than the value.

One value, two limiters. The same config value caps unauthenticated traffic per IP, to stop abuse, and authenticated traffic per user, for fairness. They want very different numbers. At 100 the abuse control was reasonable and real sessions broke. At 10000 sessions are fine and one unauthenticated address can send over a hundred requests a second. Splitting them is additive and backward compatible if the new value defaults from the existing one. We have not done it, since it changes the gateway's public configuration.

Adds a capture that can be layered onto any Playwright test. For each request to the gateway it records the time, the subject of the bearer token, the JSON-RPC method and the response status. The token itself is never stored, since these records end up in CI artifacts.

The gateway logs cannot answer how much a single user consumes: they are truncated to the last ten thousand lines and never carry the caller and the call on the same line.

Signed-off-by: nicosampler <nf.dominguez.87@gmail.com>
Portfolio tests use plain Playwright rather than the shared fixture, so they were not being measured. They now import a test that adds the capture and nothing else, since inheriting the shared fixture would also bring console tracking, popup handling and failure screenshots these tests were not written against.

Signed-off-by: nicosampler <nf.dominguez.87@gmail.com>
Reads the captures and prints the busiest minute for each user, which call dominates it, and what connecting costs before login. Also writes a JSON file so runs can be compared over time.

Reads files only, so it needs nothing running and can be pointed at CI artifacts.

Signed-off-by: nicosampler <nf.dominguez.87@gmail.com>
@nicosampler
nicosampler requested a review from a team as a code owner August 14, 2026 21:24
@mjuchli-da mjuchli-da changed the title Measure gateway request rate during e2e runs chore: measure gateway request rate during e2e runs Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Perform load tests

2 participants