perf: spanner mutation limit increase to 80000 max_total_records#2327
Open
taddes wants to merge 3 commits into
Open
perf: spanner mutation limit increase to 80000 max_total_records#2327taddes wants to merge 3 commits into
taddes wants to merge 3 commits into
Conversation
c6ed0d5 to
13101bf
Compare
13101bf to
30a52ea
Compare
30a52ea to
3693933
Compare
chenba
reviewed
May 22, 2026
| | Table | Description | | ||
| | ------------------ | ------------------------------------------------------------------------------------------------------------ | | ||
| | `user_collections` | Per-user metadata about each collection (modified time, record count, total bytes). Parent of `bsos`/`batches` via `INTERLEAVE IN PARENT`. | | ||
| | `bsos` | Stores Basic Storage Objects (BSOs) the synced records. Interleaved in `user_collections`. | |
Collaborator
There was a problem hiding this comment.
Suggested change
| | `bsos` | Stores Basic Storage Objects (BSOs) the synced records. Interleaved in `user_collections`. | | |
| | `bsos` | Stores Basic Storage Objects (BSOs), the synced records. Interleaved in `user_collections`. | |
| | `batches` | Temporary staging row per in-progress batch upload. Interleaved in `user_collections`. | | ||
| | `batch_bsos` | BSOs belonging to a batch, pending commit. Interleaved in `batches`. | | ||
|
|
||
| All `bsos` and `batches` rows are physically co-located with their `user_collections` parent Spanner's interleaving puts a user's collection metadata, BSOs, and pending batches on the same split. `ON DELETE CASCADE` from `batches` to `batch_bsos` and `user_collections` `bsos`/`batches` means parent deletes wipe descendants atomically. |
Collaborator
There was a problem hiding this comment.
Suggested change
| All `bsos` and `batches` rows are physically co-located with their `user_collections` parent Spanner's interleaving puts a user's collection metadata, BSOs, and pending batches on the same split. `ON DELETE CASCADE` from `batches` to `batch_bsos` and `user_collections` `bsos`/`batches` means parent deletes wipe descendants atomically. | |
| All `bsos` and `batches` rows are physically co-located with their `user_collections` parent. Spanner's interleaving puts a user's collection metadata, BSOs, and pending batches on the same split. `ON DELETE CASCADE` from `batches` to `batch_bsos` and `user_collections` `bsos`/`batches` means parent deletes wipe descendants atomically. |
|
|
||
| ### Env var | ||
|
|
||
| `max_total_records` is set in production via the `SYNC_SYNCSTORAGE__LIMITS__MAX_TOTAL_RECORDS` environment variable with no redeploy required to change it. The standalone-server default in `syncstorage-settings/src/lib.rs` is 10,000 and applies to non-Spanner backends; the Spanner production deployment always overrides it. |
Collaborator
There was a problem hiding this comment.
no redeploy required to change it
I'm having trouble picturing how this works. How does the running process get the new value?
Collaborator
Author
There was a problem hiding this comment.
Sry, this was in the local context when working with the config.local.toml which I altered to reflect the new value. Good catch
Collaborator
There was a problem hiding this comment.
When working locally, one needs to restart the service when the settings changed no?
Collaborator
Author
There was a problem hiding this comment.
Yup, and often just a fresh cargo build (or more scoped command depending on which db)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Much of this was understanding the underlying mechanics of mutations and updating the proper math to reflect the mutation limit capacity ok 80,000 in relation to
max_total_recordsSome details on ths: Spanner caps mutations at 80,000 per commit, and each BSO row in the batch commit costs up to 12 mutations in the worst case (4 primary-key columns + 4 non-PK columns + a delete-and-insert on each of the two secondary indexes). On top of that, the commit pays a fixed 13 mutations of overhead: 6 for the parent
user_collectionsupsert, 1 for the batch delete, and 6 to refresh quota counts. Solving 12N + 13 <= 80,000 gives a ceiling of 6,665 records, so we pick 6,656 (= 1,664 × 4) to preserve the same propotion the legacy 1,664/20,000 cap had, leaving 115 mutations of margin for breathing room.docs/src/syncstorage/syncstorage-spanner-db.mdas the canonical reference for the Spanner backend with schema, full configuration option, and per-commit mutation budget.max_total_recordsinconfig/local.example.tomlfrom 1666 to 6656 to match the value to Spanner's 80,000-mutation-per-commit ceiling.To Check
make doc-prevrenders the new page cleanly (mermaid diagram + internal links)/info/configurationreturnsmax_total_records: 6656against the bumped dev configIssue(s)
Helm Chart update here
Closes STOR-190.