Skip to content

perf: spanner mutation limit increase to 80000 max_total_records#2327

Open
taddes wants to merge 3 commits into
masterfrom
perf/spanner-mutation-limit-STOR-190
Open

perf: spanner mutation limit increase to 80000 max_total_records#2327
taddes wants to merge 3 commits into
masterfrom
perf/spanner-mutation-limit-STOR-190

Conversation

@taddes
Copy link
Copy Markdown
Collaborator

@taddes taddes commented May 21, 2026

Description

Much of this was understanding the underlying mechanics of mutations and updating the proper math to reflect the mutation limit capacity ok 80,000 in relation to max_total_records

Some details on ths: Spanner caps mutations at 80,000 per commit, and each BSO row in the batch commit costs up to 12 mutations in the worst case (4 primary-key columns + 4 non-PK columns + a delete-and-insert on each of the two secondary indexes). On top of that, the commit pays a fixed 13 mutations of overhead: 6 for the parent user_collections upsert, 1 for the batch delete, and 6 to refresh quota counts. Solving 12N + 13 <= 80,000 gives a ceiling of 6,665 records, so we pick 6,656 (= 1,664 × 4) to preserve the same propotion the legacy 1,664/20,000 cap had, leaving 115 mutations of margin for breathing room.

  • Add docs/src/syncstorage/syncstorage-spanner-db.md as the canonical reference for the Spanner backend with schema, full configuration option, and per-commit mutation budget.
  • Bumped dev max_total_records in config/local.example.toml from 1666 to 6656 to match the value to Spanner's 80,000-mutation-per-commit ceiling.

To Check

  • make doc-prev renders the new page cleanly (mermaid diagram + internal links)
  • /info/configuration returns max_total_records: 6656 against the bumped dev config
  • Ensure things look good in stage when we roll out update

Issue(s)

Helm Chart update here
Closes STOR-190.

@taddes taddes self-assigned this May 21, 2026
@taddes taddes force-pushed the perf/spanner-mutation-limit-STOR-190 branch from c6ed0d5 to 13101bf Compare May 21, 2026 19:50
@taddes taddes force-pushed the perf/spanner-mutation-limit-STOR-190 branch from 13101bf to 30a52ea Compare May 21, 2026 19:59
@taddes taddes changed the title perf: spanner mutation limit stor 190 perf: spanner mutation limit increase to 80000 May 21, 2026
@taddes taddes changed the title perf: spanner mutation limit increase to 80000 perf: spanner mutation limit increase to 80000 max_total_records May 21, 2026
@taddes taddes marked this pull request as ready for review May 21, 2026 20:00
@taddes taddes force-pushed the perf/spanner-mutation-limit-STOR-190 branch from 30a52ea to 3693933 Compare May 21, 2026 20:09
| Table | Description |
| ------------------ | ------------------------------------------------------------------------------------------------------------ |
| `user_collections` | Per-user metadata about each collection (modified time, record count, total bytes). Parent of `bsos`/`batches` via `INTERLEAVE IN PARENT`. |
| `bsos` | Stores Basic Storage Objects (BSOs) the synced records. Interleaved in `user_collections`. |
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
| `bsos` | Stores Basic Storage Objects (BSOs) the synced records. Interleaved in `user_collections`. |
| `bsos` | Stores Basic Storage Objects (BSOs), the synced records. Interleaved in `user_collections`. |

| `batches` | Temporary staging row per in-progress batch upload. Interleaved in `user_collections`. |
| `batch_bsos` | BSOs belonging to a batch, pending commit. Interleaved in `batches`. |

All `bsos` and `batches` rows are physically co-located with their `user_collections` parent Spanner's interleaving puts a user's collection metadata, BSOs, and pending batches on the same split. `ON DELETE CASCADE` from `batches` to `batch_bsos` and `user_collections` `bsos`/`batches` means parent deletes wipe descendants atomically.
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
All `bsos` and `batches` rows are physically co-located with their `user_collections` parent Spanner's interleaving puts a user's collection metadata, BSOs, and pending batches on the same split. `ON DELETE CASCADE` from `batches` to `batch_bsos` and `user_collections` `bsos`/`batches` means parent deletes wipe descendants atomically.
All `bsos` and `batches` rows are physically co-located with their `user_collections` parent. Spanner's interleaving puts a user's collection metadata, BSOs, and pending batches on the same split. `ON DELETE CASCADE` from `batches` to `batch_bsos` and `user_collections` `bsos`/`batches` means parent deletes wipe descendants atomically.


### Env var

`max_total_records` is set in production via the `SYNC_SYNCSTORAGE__LIMITS__MAX_TOTAL_RECORDS` environment variable with no redeploy required to change it. The standalone-server default in `syncstorage-settings/src/lib.rs` is 10,000 and applies to non-Spanner backends; the Spanner production deployment always overrides it.
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no redeploy required to change it

I'm having trouble picturing how this works. How does the running process get the new value?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sry, this was in the local context when working with the config.local.toml which I altered to reflect the new value. Good catch

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When working locally, one needs to restart the service when the settings changed no?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, and often just a fresh cargo build (or more scoped command depending on which db)

@taddes taddes requested a review from chenba May 22, 2026 20:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants