Repository navigation
test(soak): gate worker memory growth and add an opt-in completion stall (0.6) - #694
Merged
Merged
Conversation
…all (0.6) The persona soak samples the resident memory of the soak process into memory.csv and fails with memory_growth when the least-squares slope of resident memory over executions, after the warm-up, exceeds AWA_SOAK_MEMORY_SLOPE_MIB_PER_100K (default 64 MiB per 100k executions). AWA_SOAK_COMPLETION_STALL_SECONDS holds the queue-storage completion tables halfway through the load while claims and handlers continue; memory_growth_under_stall fails when resident memory rises by more than AWA_SOAK_COMPLETION_STALL_MEMORY_MIB over the stall. Off by default.
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
…emory-gate-0.6 # Conflicts: # awa/tests/persona_soak/config.rs # awa/tests/persona_soak/mod.rs
A stall that could not take its locks, or a run without a queue-storage schema, left memory_growth_under_stall unevaluated and the run passing.
…roducers The soak process also holds the harness ledger, which grows with every enqueued job. On a host that cannot keep up, that growth landed on few executions and failed memory_growth with no leak in the runtime. The slope is now reported as not evaluated when more than 1.25 jobs were enqueued per execution after the warm-up, and memory.csv records the ledger size.
hardbyte
marked this pull request as ready for review
November 13, 2026 02:01
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The persona soak now records the resident memory of the soak process, which hosts every Rust runtime, into
memory.csvwith the executions so far and the size of the harness ledger. After a warm-up, the least-squares slope of resident memory over executions must stay under 64 MiB per 100k executions, or the run fails withmemory_growth.The harness ledger also grows with every enqueued job, so when the runtimes fall behind the producers (more than 1.25 jobs enqueued per execution after the warm-up) the slope is reported as not evaluated rather than failed. Without this, a host short of CPU failed the gate on harness growth alone.
AWA_SOAK_COMPLETION_STALL_SECONDS(off by default) holds the queue-storage completion tables halfway through the load while claims and handlers carry on. Resident memory may rise by at mostAWA_SOAK_COMPLETION_STALL_MEMORY_MIB(default 128) over the stall, or the run fails withmemory_growth_under_stall. A stall that was requested but never held also fails the run.Stall phase status
#686 was closed unmerged, but the same finalization bound reached
release/0.6.6in 0.6.15 (23ff5bb; CHANGELOG "Worker finalization is bounded…"). It has not been shown to keep the stall gate green: on a 4-CPU host under heavy load, two 480 s soaks with a 120 s stall on the current branch rose 424 MiB (Rust personas with chaos and the bursty producer) and 185 MiB (mia, tom, sam, alex, fanout), both over the 128 MiB limit. The stall phase therefore stays off by default and should not be turned on in a release gate until it passes on a representative host.Testing
cargo fmt,cargo clippy -p awa --all-targets --all-features -D warningsclean.persona_soak_testunit tests: 68 passed against PostgreSQL 18.release/0.6.6: no stall,memory_growthpass at +30 MiB; 120 s stall on the branch before the finalization bound, +845 MiB.