Repository navigation
fix(mongodb): retry an unconditional PutItem that conflicts on a GSI table - #487
Open
LeeroyHannigan wants to merge 8 commits into
Open
LeeroyHannigan wants to merge 8 commits into
LeeroyHannigan wants to merge 8 commits into
Conversation
Amazon DynamoDB applies concurrent PutItem calls on one key one after another. A call without a condition succeeds, the last writer's item is stored, and with ReturnValues ALL_OLD each call returns the item it replaced. Measured on 2026-10-21 against the service in us-east-1 with client retries off: 32 writers x 63 rounds of racing unconditional puts on a new key per round answered 2,016 of 2,016 with 200 on hash, GSI and stream tables; 32 writers on one key for 63 rounds answered 2,016 of 2,016 on GSI and stream tables; one key with half the writers sending PutItem and half DeleteItem answered 22,400 puts and 22,400 deletes with 200 and no error (seven runs of 64 writers x 100 operations on a hash table, three runs of 32 x 200 on a GSI table); 32 writers racing PutItem against DeleteItem, UpdateItem and BatchWriteItem on one key answered every call with 200; and 64 writers with one putter per three deleters on one key answered 960 puts and 2,880 deletes with 200. In every cell the ALL_OLD images, the final item, the index and the stream records formed one serial history. tests/test_put_item_create_race.py runs against the service as well as against ExtendDB and checks the race on 8 clients. On MongoDB, two puts that create the same item could each find no document and each insert it. The second insert fails with a duplicate-key error (E11000), which findAndModify reports as a command error. The data collection has two unique indexes: _id_, and one on the key fields (pk, plus the sort key field on a table with a sort key). MongoDB retries such an upsert itself only when the filter is an equality on the fields of the index that raised the error. The put filters on pk and, on a table with a sort key, the sort key field, and every measured duplicate-key error names _id_, so the server does not retry it. The sessionless path (no condition, no index, no stream) returned the error as an internal error (HTTP 500), and the transaction path (a table with an index or a stream) classified it as fatal, which also returned an internal error. On a 16-core host the race test failed 2 or 3 of its 9 cases per run, each failure a racing put answering InternalServerError; 32 writers creating a new key per round got 3 InternalServerError responses in 2,016 puts; and one-key put and delete churn from 8 and 32 threads got 66 and 45 in 2,016 puts. Both paths now retry the put under the transient retry budget of 50 attempts with the same backoff the write-conflict retry uses. The next attempt finds the item the other put created, replaces it and returns it as the old image, so the ALL_OLD images form one chain; if a delete removed that item first, the next attempt inserts. On the transaction path the index update and the stream record take that item as the old image too. A put whose 50 attempts all lost the create race returns the internal error the PostgreSQL backend returns when its own retries of this race run out, with the attempt count in the message. is_duplicate_key now recognizes the command form as well as the write form that insert_one returns. The MongoDB xfail marker on the plain-table race tests is removed. devtools/run-mongodb-tests exports EXTENDDB_TEST_MONGODB_URI, and two storage crate tests use it, on the sessionless path and on the transaction path each: one forces every attempt to lose the race with a document that holds the item's _id under another pk and checks the give-up error after 50 attempts and that the document is unchanged; the other replaces or deletes that document 100 ms after the put starts and checks that the put succeeds, returns the other put's item or no old image, and stores its own item. Without the URI the tests print a SKIP line; with EXTENDDB_TEST_MONGODB_CONTAINER set and no URI they fail. The scratch databases are dropped when a test fails. Verified against MongoDB 7.0 as a single-node replica set. The race test passes 9 of 9 cases in each of 16 runs. The same cells as the service measurement answered every unconditional PutItem, DeleteItem and BatchWriteItem with 200 and no invariant violation; one-key put and delete churn answered 44,800 puts and 44,800 deletes with 200 (ten runs of 16 x 400 and four of 64 x 100) while 4,663 puts lost a create race and succeeded on a later attempt. With the backoff removed from the sessionless retry the lost-race-then-success test fails after 50 attempts; with a cap of 5 lost races put back on either path, with the transaction give-up error disabled, or with the transaction arm giving up at the first lost race, the storage crate tests fail; with the sessionless retry disabled the race test fails its 3 plain-table cases. One-key contention on GSI and stream tables still answers TransactionConflictException for some puts (issue 426), unchanged here. Closes #356
…table Amazon DynamoDB applies concurrent PutItem calls on one key one after another and answers each with 200; with a global secondary index the index holds the final item's entry only. Measured on 2026-10-23 against the service in us-east-1 with client retries off: 32 writers putting one key of a table with a GSI, 2,016 puts, 2,016 answered 200, and the index held the entry of the final item and no other. On MongoDB a put on a table with an index writes the item and its index entries in one transaction. MongoDB aborts the loser of a same-document conflict at once instead of queueing it, so of N writers on one key one commits per round and the rest start over, and a writer needs on the order of N attempts, more under the connection pool's queueing. The retry budget of 50 attempts ran out for a share of the puts, and the put answered the 400 TransactionConflictException main reserves for a single-item write that could not serialize against concurrent activity. The budget is now 1,000 attempts for every write path in data_engine.rs. The backoff is random, so a writer can lose many rounds in a row, and the chance of losing every attempt falls off exponentially with the budget: 400 attempts left one put in 2,016 without a commit on a host at a load average of 30 and 20 of 1,984 with 64 writers. backoff_sleep is capped at 12.8 ms, so 1,000 attempts bound a write at about 6.4 s of backoff plus the attempts' own round trips before it gives up; measured against a transaction another session holds open, a put returns after 10.5 s. A PutItem that runs out of attempts returns StorageError::Transient, which the engine answers as HTTP 503 ServiceUnavailable. The service answers a PutItem with TransactionConflictException only while a TransactWriteItems holds the item, which this backend cannot tell apart from a race against other puts; the DynamoDB error-handling documentation lists ServiceUnavailable (HTTP 503) as "OK to retry? Yes", and the SDK retry modes retry a 503 on their own, whereas a 400 is handed to the application. Measured on MongoDB 7.0 (single-node replica set, client never retries), 32 writers putting one key of a GSI table, 2,016 puts: main (9d13202) answered 303 of 2,016 with 400 TransactionConflictException; c3f8bc8 (#431) answered 322 of 2,016 with 400 TransactionConflictException; this branch answered 2,016 with 200 in each of 3 runs of the wire test, and the index held the final item's entry only. tests/test_put_item_gsi_contention.py runs this shape and fails on main and on c3f8bc8. Two storage crate tests run against the replica set named by EXTENDDB_TEST_MONGODB_URI: 32 writers putting one key on the transaction path all commit, and a put whose every attempt conflicts with a transaction another session holds open gives up with the transient error after 1,000 attempts, in 10.5 s, leaving the held item unchanged. Fixes #426
…gets TRANSIENT_RETRY_ATTEMPTS sized three different loops, and raising it to 1,000 for hot-key contention raised all three. Two of them are not contention loops and no measurement in this branch says anything about running them 1,000 times: commit_write_transaction retries commitTransaction in place after an UnknownTransactionCommitResult. The label means the commit's outcome is unknown, the commit is idempotent and the body must not be re-run, so those attempts wait for one reply rather than racing other writers. COMMIT_RETRY_ATTEMPTS keeps it at the 50 every write path used before. An unconditional PutItem that keeps losing the create race gives up with an internal error. #431 measured that path at 50 attempts with this backoff: a put blocked by a document another writer removes 100 ms after the put starts commits well inside it, and the storage tests fail with the cap put back to 5. CREATE_RACE_ATTEMPTS keeps it at 50, on the sessionless path and on the transaction path. The transaction path now gives up as soon as that budget is spent instead of carrying lost races through the contention budget. TRANSIENT_RETRY_ATTEMPTS stays at 1,000 and now sizes the contention loops only: put, delete, update, transact and the GSI backfill. The constant's doc comment said 1,000 attempts "bound the wait at about 6.4 s of backoff". backoff_sleep draws each sleep uniformly from 0 to the current base, capped at 12.8 ms, so 6.4 s is the mean of the sleeps and 12.8 s is the bound, before the attempts' own round trips. Measured on MongoDB 7.0 as a single-node replica set, a put that loses all 1,000 attempts to a transaction another session holds open answered in 9.46 s, 9.85 s and 10.00 s over three runs, where each attempt also waits out MongoDB's lock timeout. The comment now states the mean, the bound and the measurement, and the test prints the elapsed time it already measured. The RFC's write-conflict paragraph said is_transient_write_conflict returns true for the UnknownTransactionCommitResult label. It does not, by design: under that label the commit may already have applied, so the body must not be re-run, and classify_commit_error handles it alone. The paragraph now also names the commit and create-race budgets and carries the corrected mean and bound. Issue #426
main added four fields to MongoEngine in #384. The with_scratch_engine test helper builds the engine with a struct literal, so against a main that carries #384 it failed to compile with E0063. It now sets the four fields the way backup_live_tests.rs does. The comments on the retry budget give the measured wall time of a put that loses every attempt as a range from six runs on a shared host.
The 1,000-attempt budget bounds attempts, not the time a caller waits. At 256 writers putting one key of a GSI table, 89 of 5,120 calls were still waiting when the client read timeout fired at 120 s, past the SDKs' default read timeouts. The contention loops of PutItem, DeleteItem, UpdateItem and TransactWriteItems now read CONTENTION_DEADLINE (10 s) before each attempt after the first. A single-item write that reaches it answers ServiceUnavailable (HTTP 503), which the SDKs retry. A transaction that reaches it answers TransactionCanceledException with a TransactionConflict reason per item, as it does when its attempts run out and as the service answers a transaction that loses to contention. The GSI backfill loop runs in the background worker and takes no deadline.
The deadline and the 1,000-attempt budget take about the same time against a contended key, so a test could not say which bound ended a write. The engine now carries the deadline, set to CONTENTION_DEADLINE in every production construction, and a storage test sets it: 2 s makes the deadline the bound, 600 s makes the attempt budget the bound. Three paths that had no test now have one. A contended PutItem, DeleteItem and UpdateItem that reach the deadline each answer the transient error the engine answers as HTTP 503, with the operation and the deadline in the message. A TransactWriteItems that reaches the deadline answers a canceled transaction with a TransactionConflict reason per operation, which is what the service answers a transaction that loses to contention. The test named for the attempt budget reaches the attempt budget again and asserts the 1,000-attempt message rather than either message. Each test is bounded to the deadline it sets, so a loop that stopped reading the deadline fails it. The doc comment on the deadline said both bounds answer the same HTTP 503, which holds for PutItem and TransactWriteItems only. On this branch DeleteItem and UpdateItem answer 503 on the deadline and 400 TransactionConflictException when their attempt budget runs out first, measured in 2 of 4 rounds against a document a transaction held. The comment and the RFC paragraph now say that, and name the change that gives all three single-item writes one answer.
… has tests/test_put_item_gsi_contention.py asserted that all 2,016 puts of 32 writers on one key answer 200. The retry deadline this PR adds makes that false under load: a put that has been losing for ten seconds is shed with 503 ServiceUnavailable. The file failed 1 of 4 runs in the shape CI uses, two xdist workers on one server, and failed with 240 of 2,016 shed when another hot-key load shared the server. The test now asserts the contract the retry loop gives. Every call answers 200 or 503 ServiceUnavailable, never 400 TransactionConflictException and never 500; the stored item is one writer's item, whole, carrying the gk that writer sent with that w; and the index holds exactly one entry for it. The number shed depends on the load the server is under, so the test reports it instead of asserting it. The docstring keeps what the service answered to this shape, 200 to every call, and says where ExtendDB diverges. docs/differences-from-dynamodb.md gets a row for that divergence under Capacity and Throttling, with the measurements at 64 and 256 writers and what each operation answers at each bound.
LeeroyHannigan
requested review from
amrith,
c33howard,
jcshepherd,
pdf-amzn,
robinnsc and
yesyayen
as code owners
October 24, 2026 05:29
This was referenced Oct 24, 2026
robinnsc
reviewed
Oct 30, 2026
robinnsc
left a comment
robinnsc
left a comment
Collaborator
There was a problem hiding this comment.
The single-item fix holds up. On a local MongoDB 7 with a GSI + stream table and 32, 64 and 256 writers on one key, PutItem only ever answered 200 or 503. The stored item, its single index entry and the stream (one record per commit, none for a shed put, an unbroken old-image chain) were consistent after every run. UpdateItem, DeleteItem and TransactWriteItems on the same key were consistent too, and conditional puts weren't retried into wrong answers. The longest single-item call was 11.8 s at 256 writers, against 28 s on main. Uncontended puts cost the same as before. The 9 new storage tests pass, and #431's create-race fix holds (2,880 of 2,880 racing creates answered 200).
Two things I'd look at before merging, both in the first two inline comments: BatchWriteItem, and the merge with current main not compiling in test builds.
Smaller notes:
- On a GSI + stream table, writes to different keys that conflict while writing the index or the stream counter still answer 500, because those errors map to
Internalinsync_indexes_in_sessionand the stream helpers. A mixed BatchWriteItem + PutItem run gave 68 of 240 batches and 9 of 240 puts as 500 on head (main is similar). If that's #437's scope, it might be worth saying so, since the description says no 400 or 500 in any run. - A hot key keeps mongod at about 2 cores for as long as the contention lasts (about 1,800-1,900 transactions/s at 64 and 256 writers), since the backoff cap is still 12.8 ms and losers now retry for up to 10 s. That seems like a fair trade-off, just worth a sentence.
- The overshoot past 10 s (up to 12.4 s at 256 writers) comes from the attempt in flight waiting on the 50-connection pool and the commit retries. The doc comment's "deadline plus one attempt" covers it, but the attempt can be a couple of seconds on its own.
- Small doc nits: the orphaned doc comment on
fn hash_keyin the test module; the RFC decision-table row atdocs/rfcs/0000-mongodb-backend.md:238still says theUnknownTransactionCommitResultlabel is detected, which the rewritten paragraph now contradicts;contention_deadline_exhaustedprintsas_secs(), so a sub-second test deadline reads "0 seconds".
|
|
||
| let tx_options = self.write_transaction_options(); | ||
| let mut lost_create_races = 0; | ||
| let contention_started = std::time::Instant::now(); |
Collaborator
There was a problem hiding this comment.
The clock starts per storage call, and batch_write_item.rs:127-138 / :181-191 call put_item / delete_item once per item, so a BatchWriteItem gets 10 s per item. With items held and released 7-9 s apart, a 10-item batch answered 200 after 70 s, and with botocore's default 60 s read timeout the client timed out with 8 of 10 written. Main answered 400 in 0.4 s.
The bigger effect is the 503 itself. Earlier items in the batch have already committed when a later one gives up, and since 503 is retried by the SDK (main's 400 wasn't), the whole batch is resent. Repro with [delete A, put B], B held for 14 s, and another client putting A=v2 at 5 s, which got 200: the batch's resend deleted A again, so the final state is A absent, with stream records REMOVE v0, INSERT v2, REMOVE v2. With [put A=v1, put B] the final state was A=v1 after the other client's v2. On main the same shapes answered 400 once, and v2 survived.
DynamoDB returns items it couldn't write in UnprocessedItems, so the SDK never resends the applied ones. Could consider carrying one deadline per request into the batch, and returning the item that gave up plus the rest as UnprocessedItems rather than failing the batch. ImportTable (import_export.rs:154) has the same per-item shape, though there it's a fresh table.
| /// construction uses. A test sets it short to make the deadline the bound | ||
| /// a contended write reaches, or long to make the attempt budget the | ||
| /// bound, instead of racing the two. | ||
| pub(crate) contention_deadline: std::time::Duration, |
Collaborator
There was a problem hiding this comment.
git merge-tree against current main is clean, but cargo check -p extenddb-storage-mongodb --tests on the merged tree fails with E0063: missing field contention_deadline at stream_records_live_tests.rs:46. That literal came in with #543 after this branch merged main. It's the same kind of break as the #384 one the description mentions, and #431 on its own also fails against main (its with_scratch_engine literal). A single test constructor for MongoEngine might save the next round of these.
| # the server is under, so it is reported rather than asserted; what is | ||
| # asserted is that a shed put is the only failure a caller can get. | ||
| print(f"{len(failures)} of {total} puts were shed, {total - len(failures)} committed") | ||
| assert set(failures) <= {SHED}, ( |
Collaborator
There was a problem hiding this comment.
This only checks the error class, so any build that answers 503 at the first conflict passes as long as one put commits. That includes the old budget of 50 with the new give-up. The storage tests do pin the budget (the deadline tests and the 256-writer test fail with 50), so it's not unguarded, but a loose floor here, say at least half committed, would give the wire test some teeth without the 0-shed flakiness the description mentions. Similarly, unconditional_put_that_spends_its_attempt_budget_gives_up_with_a_transient_error builds its expected message from TRANSIENT_RETRY_ATTEMPTS and only bounds time from above, so it can't tell which budget ran.
| kwargs["endpoint_url"] = endpoint_url | ||
| if endpoint_url.startswith("https://"): | ||
| kwargs["verify"] = False | ||
| return boto3.client(**kwargs) |
Collaborator
There was a problem hiding this comment.
nit: this module-scoped client is shared by all 32 writer threads. A client per thread would avoid the occasional TLS error a shared botocore client gives under threads. Here that would fail the test rather than pass it, so it's a flake risk, not a false pass.
| /// 503 whichever bound it reaches. | ||
| /// | ||
| /// The deadline is read before an attempt starts, so a request answers within | ||
| /// the deadline plus one attempt. It governs the loops that serve a request: |
Collaborator
There was a problem hiding this comment.
The TTL worker reaches one of these loops too: ttl_worker.rs:474 calls DataEngine::delete_item with a condition, so on a GSI or stream table it takes the transaction loop with the deadline. A contended expired item can now hold the sweep for up to 10 s (or 1,000 attempts) where it was 50 attempts before. That's probably fine, but the comment says only request loops take the deadline.
| // this loop does. The error means another put's item is already in | ||
| // the collection, so the next attempt finds that item, replaces it | ||
| // and returns it as the old image; if a delete removed that item | ||
| // first, the next attempt inserts. Attempts share the transient |
Collaborator
There was a problem hiding this comment.
nit: since 544103c the create race has its own CREATE_RACE_ATTEMPTS budget, so "Attempts share the transient retry budget" here and "uses the same retry budget and backoff as a write conflict" near line 563 are out of date.
| | Provisioned throughput | Token bucket per table/partition | Token bucket per table/partition, matching DynamoDB's burst and refill behavior | | ||
| | On-demand capacity | Automatic scaling | Fixed initial burst capacity (4000 WCU / 12000 RCU), no auto-scaling | | ||
| | Throttling | Always on; throttles requests that exceed provisioned/burst capacity. No setting to disable | Configurable via `throttling_enabled` runtime setting (default: `true`) | | ||
| | Sustained contention on one item (MongoDB backend) | Applies concurrent single-item writes on one key one after another and answers 200 to each. Measured to every call at 32 and 64 writers putting one key of a table with a GSI. A `PutItem` is answered 400 `TransactionConflictException` only while an in-flight `TransactWriteItems` holds the item (33 of 80 calls measured), and a `TransactWriteItems` that loses to other transactions on one key is answered 400 `TransactionCanceledException` with a `TransactionConflict` reason (107 of 160 calls measured) | A write on a table with a secondary index writes the item and its index entries in one transaction, so concurrent writes on one key abort each other and are retried. One request's retry loop stops after 10 seconds (`CONTENTION_DEADLINE`) or 1,000 attempts, whichever comes first, and the request is then shed instead of waiting out the queue of writers on that key: `PutItem` answers 503 `ServiceUnavailable` on either bound, which the SDK retry modes retry and which succeeds once the contention has passed. `DeleteItem` and `UpdateItem` answer 503 on the deadline and 400 `TransactionConflictException` when their 1,000 attempts run out first. `TransactWriteItems` answers `TransactionCanceledException` with a `TransactionConflict` reason per item on either bound, which is the answer the service gives. Measured with one GSI table and all writers on one key: at 64 writers all 1,280 puts committed and none were shed, with the longest call 9.7 s; at 256 writers 1,262 of 5,120 committed and 3,858 were shed, with the longest call 11.7 s. Without the deadline the same 256-writer shape left calls waiting 99.5 s, so a client sees a shed request where it used to see a request outlive its own read timeout. The other three backends keep writes on one key waiting rather than shedding them, and the deadline does not apply to them | |
Collaborator
There was a problem hiding this comment.
"at 64 writers all 1,280 puts committed and none were shed" didn't hold on a moderately loaded host here: 106 and 149 of 1,280 were shed at load averages 6.8 and 10.5. The description already says the shed count depends on load. Might be worth saying the same in this user-facing row, or keeping only the 256-writer bound.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #426. Stacked on #431 (
fix/mongodb-putitem-create-race, head c3f8bc8):this branch's first commit is #431's, and it should merge after it. The code in
this PR does not call #431's create-race code, but its storage tests read
EXTENDDB_TEST_MONGODB_URI, whichdevtools/run-mongodb-testsexports onlybecause #431 added that line. Merged on its own, those tests would print a SKIP
and pass without running.
On MongoDB a put on a table with an index writes the item and its index entries
in one transaction. MongoDB aborts the loser of a same-document conflict at
once instead of queueing it, so of N writers on one key one commits per round
and the rest start over, and a writer needs on the order of N attempts, more
under the connection pool's queueing. A budget of 50 attempts ran out for a
share of the puts, and the put answered the 400 TransactionConflictException
that main reserves for a single-item write that could not serialize.
Two bounds now end a contended write, and both answer an error the SDKs retry.
The contention budget is 1,000 attempts, and a PutItem that spends it returns
StorageError::Transient, answered as HTTP 503 ServiceUnavailable. A request'sretry loop also stops after
CONTENTION_DEADLINE, 10 seconds, because theattempt budget bounds attempts and not the time a caller waits: at 256 writers
on one key, 89 of 5,120 calls were still waiting when the client's 120 s read
timeout fired. With the deadline the longest of 6,400 calls on that shape was
11.7 s.
Merged with main
This branch's base predates #384, which added four fields to
MongoEngine(backup_liveness,restore_server_slot_count,restore_server_slots,restore_account_slots). Thewith_scratch_enginetest helper that #431 added builds aMongoEnginewith a struct literal, so against a main that carries #384 the tree is conflict free and rustc rejects it with E0063. The branch now merges origin/main, and the helper sets the four fields the waybackup_live_tests.rsdoes. #431 on its own fails the same way against the current main.What the service does
Measured against DynamoDB in us-east-1 with client retries off
(
retries={"max_attempts": 0}), on existingbb-tables with a string hashkey, a GSI and a stream, on 2026-10-23 and 2026-10-24.
ADD #c :one SET #g = :gwith ALL_NEW on one keySET l = list_append(l, :m)on one keyThe service sheds none of the single-item load at any writer count measured,
including 64 concurrent writers on one key. It answers a single-item write with
TransactionConflictExceptiononly while aTransactWriteItemsholds the item,which this backend cannot tell apart from a race against other puts. It does
shed concurrent transactions on one key, with the same
TransactionCanceledExceptionandTransactionConflictreason this backendanswers when a transaction stops retrying.
What each bound answers
Measured on this branch with another session holding the contended item inside
a MongoDB transaction it never commits, so every attempt of the retry loop
conflicts. Four writes ran at once on that key, client retries off, four
rounds, host load 11 to 13:
PutItem answers 503 on either bound, and
TransactWriteItemsanswersTransactionCanceledExceptionwith aTransactionConflictreason per item oneither bound. DeleteItem and UpdateItem answer 503 on the deadline and 400
TransactionConflictExceptionwhen their 1,000 attempts run out first, whichthe delete did in 2 of the 4 rounds above: the two bounds take about the same
time on this shape, so which one the loop reaches decides which error class the
caller is told. Those two 400s are what main answers for every spent budget on
those paths, and #437 is the change that routes them through the same transient
give-up. After #437 merges, every single-item write answers 503 on either
bound. The deadline moves the common case on those two paths to 503, so the
400 is reached less often than on main.
A client on an SDK retry mode retries the 503 and the retry commits once the
contention has passed. The message the give-up carries ("PutItem: the write
lost to concurrent writers for 10 seconds; retry the request") reaches the
server log; the client gets the fixed 503 body every transient error on this
backend answers with.
What each status means here
10 s deadline. The error-handling documentation lists ServiceUnavailable
(HTTP 503) as "OK to retry? Yes", and the SDK retry modes retry it, whereas a
400
TransactionConflictExceptionis handed to the application.operation: a
TransactWriteItemsreached either bound. This is what theservice answers a transaction that loses to contention on one key, measured
in 107 of 160 calls.
an item another writer had created and removed again. That is fix(mongodb): retry an unconditional PutItem that loses the create race #431's answer,
the same one the PostgreSQL backend gives when its own retries of this race
run out.
One constant sized three loops, and which loops take the deadline
TRANSIENT_RETRY_ATTEMPTSsized three different loops, so raising it to 1,000raised all three. Two of them are not contention loops:
TRANSIENT_RETRY_ATTEMPTS = 1000now sizes the contention loops only: put,delete, update, transact and the GSI backfill. Set by the measurements in
its doc comment: 50 attempts left 15 to 25 percent of the puts of 32 clients
on one key without a commit, 400 left one put in 2,016 on a loaded host.
COMMIT_RETRY_ATTEMPTS = 50sizescommit_write_transaction's in-placeretry after an
UnknownTransactionCommitResult. That label means thecommit's outcome is unknown, the commit is idempotent and the body must not
be re-run, so those attempts wait for one reply rather than racing other
writers. No measurement here sizes a longer wait, and a longer one only
delays the HTTP 500 the path answers when the outcome stays unknown.
CREATE_RACE_ATTEMPTS = 50sizes the create-race give-up, on thesessionless path and on the transaction path. fix(mongodb): retry an unconditional PutItem that loses the create race #431 measured that budget with
this backoff: a put blocked by a document another writer removes 100 ms
after the put starts commits well inside it, and fix(mongodb): retry an unconditional PutItem that loses the create race #431's storage tests fail
with the cap put back to 5. The transaction path now gives up as soon as
that budget is spent, rather than carrying lost races through the contention
budget.
CONTENTION_DEADLINE = 10 sbounds the four loops that serve a request: put,delete, update and transact. It is read before an attempt starts and never
before the first one, so a request answers within the deadline plus the attempt
already running, and no request answers 503 without having tried the write
once. The GSI backfill loop keeps the attempt budget and takes no deadline: it
runs in the background worker, where no caller is waiting for a reply and
giving up would leave the index CREATING. The sessionless PutItem fast path and
commit_write_transactiontake no deadline either, because neither is acontention loop.
The engine carries the deadline as a field set to
CONTENTION_DEADLINEinevery production construction, which is what lets a test decide which bound it
exercises instead of racing 1,000 attempts against a wall clock.
How long each bound takes
backoff_sleepdraws each sleep uniformly from 0 to the current base, cappedat 12.8 ms, so 1,000 attempts sleep about 6.4 s in total on average and 12.8 s
at worst, before the attempts' own round trips.
Measured on MongoDB 7.0 as a single-node replica set, against a transaction
another session holds open, where each attempt also waits out MongoDB's lock
timeout:
What the bounds buy and what they cost
MongoDB 7.0 single-node replica set, client retries off, a fresh GSI table per
run, all writers on one key, 20 puts each. The server was the only extenddb
process on the host; the host itself was shared. Host load average 5.6 to 22.3
for the budget-only columns and 5.0 to 15.8 for this head, so the two sets are
not from an idle host and are not from the same load.
At 64 writers on one key the raised budget turns 943 shed calls into commits,
and the deadline is never reached, so nothing is shed and the longest call
falls from 28.0 s to 9.7 s. At 256 writers the deadline turns 3,471 calls that
used to commit after a long wait into 503s the SDK retry modes retry, and the
commit rate is 6.9 against 5.4 on a different host load. In both runs the
stored item was the last writer's and the index held one entry for its indexed
value.
The same 256-writer shape against this head's parent commit, which has the
1,000-attempt budget and no deadline, on the same host and container, 5 rounds
each: without the deadline p50 20,983 ms, p99 98,856 ms and the longest call
99,534 ms; with it p50 10,079 ms, p99 10,662 ms and the longest call 11,658 ms.
Those two runs sent 5 rounds and 20 rounds, so their commit counts measure
different amounts of work; the per-call latencies are measured on the same
shape and the same host. A client on botocore's default 60 s read timeout gets
an answer it can retry on this shape instead of a socket timeout.
The GSI backfill keeps its attempt budget and no deadline
A table with 600 items, 48 writers putting one of those keys continuously with
client retries off, then
UpdateTableadding a GSI ong: the index reachedACTIVE 22.6 s after the
UpdateTable, which is past the 10 s deadline, andheld 600 entries for the 600 base items. The 48 concurrent writers answered
3,038 times with 200 and twice with 503. So the request path's deadline does
not stop the background backfill, which is why that loop reads no clock.
The known limit this PR does not change
The hot-key commit rate on this backend, measured above on a shared host, is
2.4 to 6.5 commits per second at budget 50, 5.4 to 25.9 at budget 1,000, and
6.9 to 38.9 at budget 1,000 with the deadline. The service answered 1,920 calls
from 64 writers on one key in 2.5 s, about 770 per second. That comes from
aborting the loser of a same-document conflict instead of queueing it, which
neither this PR nor #437 touches. Each changes which error a losing write
receives, not the rate at which writes commit.
Tests
tests/test_put_item_gsi_contention.py: 32 writers putting one key of atable with a GSI, 2,016 puts, client retries off. Every call must answer 200
or 503 ServiceUnavailable, never 400 TransactionConflictException and never
500; the stored item must be one writer's item, whole, carrying the
gkthatwriter sent with that
w; and the index must hold exactly one entry for it.How many calls are shed depends on the load the server is under, so the test
reports that count instead of asserting it. An earlier revision of this file
asserted that every call answers 200, which the deadline makes false under
load: it failed 1 of 4 runs in the shape CI uses, two xdist workers on one
server, and failed with 240 of 2,016 shed when another hot-key load shared
the server. The file also runs on the PostgreSQL and SQLite integration jobs,
which shed nothing on this shape.
docs/differences-from-dynamodb.mdcarries a row for the divergence underCapacity and Throttling: what each operation answers at each bound, and the
measurements at 64 and 256 writers.
EXTENDDB_TEST_MONGODB_URI. 32 writers putting one key on the transactionpath all commit. A contended PutItem, DeleteItem and UpdateItem that reach
the deadline each answer the transient give-up naming the operation and the
deadline. A contended
TransactWriteItemsthat reaches the deadline answersa canceled transaction with a
TransactionConflictreason per operation, andneither operation is applied. A put that spends its 1,000 attempts answers
the give-up naming the attempts, with the held item unchanged. Each of those
tests sets the engine's deadline, 2 s to make the deadline the bound and
600 s to make the attempt budget the bound, so none of them races the two.
always runs, no attempt starts once the deadline has passed, a deadline a
test sets decides the answer, and the give-up carries the deadline that
passed.
named separately, and still report the give-up after 50 attempts on both
paths.
What fails without this branch
On MongoDB 7.0, 32 writers putting one key of a GSI table, 2,016 puts per run,
client retries off:
Each of the four storage tests of the two bounds was run against a broken
version of the path it covers, and each failed: with the loop reading the
production constant instead of the engine's deadline, all three deadline tests
and the attempt-budget test fail; with the deadline check removed from the
transact loop, the transaction test fails on its elapsed bound; with either
give-up returning
TransactionConflictinstead ofTransient, the test ofthat give-up fails on the error class.
Gates
cargo +1.97.0 fmt --all -- --check: clean.cargo +1.97.0 clippy --all-targets -- -D warnings: no warnings.cargo test --workspace --exclude extenddb-storage-cassandrapluscargo test -p extenddb-storage-cassandra --lib, on stable and on 1.88.0:1,837 passed, 0 failed, 26 ignored on each, with
EXTENDDB_TEST_MONGODB_URIset so the URI-gated storage tests ran rather than skipped. The Cassandra
integration targets ran as unit tests only, because another process on the
build host holds a Cassandra on 127.0.0.1:9042 that those targets would
otherwise use. Nothing in this branch touches Cassandra code.
test_put_item_create_race, test_batch_operations and test_item_operations:
171 passed, 0 failed, 1,164 deselected. (The filter also named
test_stream_write_contention, which is MongoDB: a write conflict on the stream sequence counter answers 500, and BatchWriteItem can answer 500 after one of its puts committed #437's file, and test_transactions,
which matches no file; neither selected a test.)
tests/test_put_item_gsi_contention.pyin the shape CI uses,two xdist workers over four files on one server: 10 of 10 passed, and the
server shed no put in any of them. With 128 writers on another table's hot
key sharing the server, the same test passed with 117 of 2,016 puts shed,
which is the condition the earlier assertion failed under.
server: the contention test passes on both. Neither backend sheds a put on
this shape.
Cargo.lockunchanged, so the license notice checks do not apply.