chore: measure gateway request rate during e2e runs - #2288
Open
nicosampler wants to merge 4 commits into
Open
Conversation
Adds a capture that can be layered onto any Playwright test. For each request to the gateway it records the time, the subject of the bearer token, the JSON-RPC method and the response status. The token itself is never stored, since these records end up in CI artifacts. The gateway logs cannot answer how much a single user consumes: they are truncated to the last ten thousand lines and never carry the caller and the call on the same line. Signed-off-by: nicosampler <nf.dominguez.87@gmail.com>
Portfolio tests use plain Playwright rather than the shared fixture, so they were not being measured. They now import a test that adds the capture and nothing else, since inheriting the shared fixture would also bring console tracking, popup handling and failure screenshots these tests were not written against. Signed-off-by: nicosampler <nf.dominguez.87@gmail.com>
Reads the captures and prints the busiest minute for each user, which call dominates it, and what connecting costs before login. Also writes a JSON file so runs can be compared over time. Reads files only, so it needs nothing running and can be pointed at CI artifacts. Signed-off-by: nicosampler <nf.dominguez.87@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Closes #1202.
Every e2e test now records the requests its browser sends to the wallet gateway. A script turns those recordings into a report. Nothing in the gateway changes.
Comments below are added for this description, the script does not print them.
Three things to keep in mind about the numbers:
What we implemented
core-wallet-test-utilsthat can be added to any Playwrighttest. It records the time, thesubof the bearer token, the JSON-RPC method and the response status. The token itself is never stored.Alternatives we discarded
Gateway logs from CI. Truncated to the last ten thousand lines, which at debug level is twenty seconds to two minutes of a run. They also never carry the caller and the call on the same line, so traffic cannot be attributed to a user.
Playwright traces. They do contain everything we need. But CI only records them on a retry, so a passing run leaves nothing to read, and always recording them would grow artifacts that already reach tens of megabytes per job.
An access log in the gateway. The only source that counts exactly what the limiter counts, and useful beyond this ticket. It is production code and changes how the gateway behaves in operation, so it is your call rather than ours.
Notes
On the ticket's premise. The default went from 100 to 10000 in January. Back then portfolio had one test file and no polling, and was already going over 100. It now has seventeen tests and refetches every query every five seconds, and reaches 2366. Nobody touched the limiter in between. A number picked today will go stale the same way, which is why the reproducible measurement matters more than the value.
One value, two limiters. The same config value caps unauthenticated traffic per IP, to stop abuse, and authenticated traffic per user, for fairness. They want very different numbers. At 100 the abuse control was reasonable and real sessions broke. At 10000 sessions are fine and one unauthenticated address can send over a hundred requests a second. Splitting them is additive and backward compatible if the new value defaults from the existing one. We have not done it, since it changes the gateway's public configuration.