Skip to content

fix(streams): disabling a stream closes its shards; enabling makes a new one - #515

Open
LeeroyHannigan wants to merge 15 commits into
mainfrom
fix/482-stream-disable-closes-shards
Open

LeeroyHannigan wants to merge 15 commits into
mainfrom
fix/482-stream-disable-closes-shards

Conversation

@LeeroyHannigan

@LeeroyHannigan LeeroyHannigan commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

UpdateTable StreamEnabled=false left the stream ENABLED with every shard open on all four backends, and a later enable reused the same stream ARN, label and shard ids (#482). Measured on the service: a disable closes the stream (DISABLED, every shard with an EndingSequenceNumber, DescribeTable without StreamSpecification, LatestStreamArn still naming it), an enable afterwards creates a new stream with a later label and four new shards, both streams stay listed, and enabling an enabled table or disabling one with no stream is refused.

A table now has a sequence of streams. Each stream's identity is carried in its shard ids (shardId-{table_id}-{label_millis}{view_type}{index}), so no backend needs a schema change, a migration or a catalog version bump; ids written before this scheme, including the shardId-{table_name}-{index} form from before v0.1.14, belong to the table's current stream until its first enable under this release. The rules shared by the four backends (grouping shards into streams, status, paging, the refusals) live in extenddb_storage::stream_lineage; each backend has a stream_lifecycle.rs that reads its rows and calls them.

What a client sees

Every sentence below was measured on the service (two repeats each) and is answered byte for byte on all four backends:

  • Disable: DescribeStream reports DISABLED with the stream's own view type; each shard's EndingSequenceNumber is its last record's sequence number, or its starting number when empty; DescribeTable and the UpdateTable response omit StreamSpecification; LatestStreamArn keeps the closed stream. Writes record only to open shards.
  • Enable: a new stream with a later label and four new shard ids; ListStreams lists both, newest first; a shard id of one stream on the other's ARN is Requested resource not found: Shard does not exist.
  • Refusals: Table already has an enabled stream: TableName: <name>; Table has no stream to disable: TableName: <name>; a stream change beside BillingMode or ProvisionedThroughput (You cannot modify stream status while updating table IOPS), beside GlobalSecondaryIndexUpdates (You cannot create or delete index while changing stream status), beside DeletionProtectionEnabled (DeletionProtection modification must be the only operation in the request), nothing applied; a disable carrying a view type (If stream is being disabled, then UpdateViewType must not be specified); CreateTable with a stream enabled and no view type (Table is being created with a stream enabled, UpdateViewType is required); a StreamSpecification object without StreamEnabled (the model's null-member sentence); a table created with {StreamEnabled: false} stores nothing of it.
  • Paging and constraints: ListStreams carries LastEvaluatedStreamArn whenever Limit cut the page, refuses a start ARN of another table, Limit under 1 and a start ARN under 37 characters; GetShardIterator refuses a ShardId under 28 or over 65 characters and a SequenceNumber under 21, answers AT_SEQUENCE_NUMBER without a number with SequenceNumber cannot be null and an unknown stream ARN with Stream does not exist; DescribeStream refuses a short ExclusiveStartShardId and answers an unknown stream of an existing table with Stream: <arn> not found.

Consistency under concurrency

  • A write racing a disable: on the service every write accepted before the stream reports DISABLED lands before the ending number. On PostgreSQL the write path reads the open shards without a lock, locks the one shard its record goes to FOR KEY SHARE while re-checking that it is still open, and the close locks the open rows FOR UPDATE before taking the ending numbers, so a raced write lands before the ending number or records nothing; on Cassandra a new stream starts one above the larger of the enabling server's next clock value and the last ending number among the table's shards, so on one server a raced record of the closed stream falls in neither stream rather than in the new one; with several servers, a record written by a server whose clock is ahead of the enabling server's can land in the new stream (part of the cross-instance consistency item in the design document). Measured with twelve writer threads under the earlier lock shape: 0 records past the ending number and 0 in the next stream on both (before: 5 of 837 and 1); a storage test now runs eight writers against a disable on PostgreSQL and checks that no record is past its shard's ending number. MongoDB keeps a one-transaction window (see below).
  • Two overlapping stream changes: on MongoDB and Cassandra the catalog write is conditioned on the specification and label it read, so exactly one applies and the other is refused (20 of 20 pairs, before: 15 and 17 of 20 both applied). An enable re-reads the row after inserting its shards and closes them when a disable got in between, so a DISABLED stream has every shard closed (0 of 60 rounds on PostgreSQL and 0 of 30 on MongoDB; before: 11 of 30 and 2 of 30).
  • A disable closes only the stream it decided on, so a disable applying after an overlapping enable does not close the new stream; a PostgreSQL storage test applies such a disable's shard changes after the enable and checks the new stream's four shards stay open. On Cassandra a close writes a shard's ending number only when the shard has none (IF ending_sequence_number = null), so a disable whose close runs after an overlapping enable started the next stream cannot raise the closed shard's ending number above the new stream's start.
  • A stream disabled by an earlier release (stored {StreamEnabled: false}, four open shards), or left so by a close that failed after its catalog write, keeps recording writes. A disable on that table is refused as before and closes the current label's open shards first; an enable closes them too. Storage tests on SQLite and PostgreSQL seed that state. On PostgreSQL, MongoDB and Cassandra an enable closes the open shards of every earlier stream of the table, legacy rows included, so a stream left open after two failed closes in a row (a disable's, then the next enable's) is closed by the enable after that; a PostgreSQL storage test seeds that state. SQLite closes every open shard of the table in the transaction that writes the catalog row.

Compatibility

A stream created by an earlier release keeps working: DescribeStream, ListStreams, GetShardIterator, GetRecords and writes before any change; a disable closes it and keeps its view type; an enable replaces it with a labelled stream. Pinned by a PostgreSQL storage test that seeds the pre-v0.1.14 rows. One limitation: such a stream's label lived only in the table row, so after a disable and a re-enable its ARN no longer resolves: DescribeStream answers Stream: <arn> not found, ListStreams omits it, and an iterator still reading it answers Stream does not exist, so drain such a stream before enabling it again (a disable alone keeps it describable). Recording it would need a per-table stream history column and a migration on three backends; documented in the differences table.

There is no catalog version gate, so a server still on the earlier release writes records past a closed shard's ending number and re-enables a stream under its old ARN. The upgrade manual says to upgrade every server before changing a table's stream, and that rollback is a restore of the pre-upgrade database.

DescribeBackup for a backup taken after a disable keeps showing StreamDescription: {StreamEnabled: false}, as before; the view type the backends now keep for the closed stream is not added there, since the service's answer for that backup is not measured.

Re-cut onto #543

#543 (877ef39) resolves the shard against its stream inside the GetRecords read, with the table's one label deciding ownership. This branch is re-cut on top of it:

  • One set of not-found helpers, main's in extenddb_storage::util. GetShardIterator with a shard of the table's other stream, or a well-formed shard id the stream never had, answers Requested resource not found: Shard does not exist (both measured 2026-10-24); main answered Stream: <arn> not found. with a period there.
  • util::shard_in_stream reads the shard id: a labelled id belongs to the label it carries, and a legacy id to the table's current label while the table has no labelled shard. From a table's first enable under this release every later stream is labelled, which makes legacy rows beside a labelled row the earlier, closed stream. So a closed stream stays readable under its own ARN after the table has a newer one, and a token pairing the newer ARN with an older stream's shard id, labelled or legacy, is refused, as GetShardIterator refuses the same pairing. The labelled-shard check runs only for a legacy id: an EXISTS on stream_shards on SQLite and PostgreSQL (byte order, COLLATE "C" on PostgreSQL), a find_one on the {table_id, shard_id} index on MongoDB, the table's shard rows on Cassandra; a labelled id keeps fix(streams): resolve the shard against its stream inside the GetRecords read #543's statements. Unit tests on the rule, SQLite engine tests on a re-enabled table and on a table whose legacy stream was closed by a disable and replaced by an enable; the wire fixture's reread after the re-enable covers the first on every backend.
  • Cassandra get_stream_records keeps main's ownership check (the shard row's table id and shard_in_stream) and its two round trips, reading the records from the shard's physical partition in parallel with the shard row and applying the shard's sequence range to the rows it gets back. One stream_table_row.
  • Main's wire test that pinned a re-enable to the same ARN moves to the new ARN the service answers; main's PostgreSQL and Cassandra ownership tests pin a label by moving the shard rows to the ids that label gives them, since a shard id carries its stream's label.
  • The engine keeps main's constraint helper beside the shard id length constants.

Review changes

  • GetShardIterator applies the 65-character ShardId maximum only when the shard is not found, so the shardId-{table_name}-{index:016} shards of a table named with more than 40 characters stay readable.
  • A StreamSpecification that is not an object keeps SerializationException from deserialisation; the null-member check applies to objects.
  • PostgreSQL writes lock one shard row FOR KEY SHARE instead of all four FOR SHARE (above).
  • Cassandra's new stream starts one above the larger of the enabling server's clock value and the closed streams' ending numbers (above); the design document's two sentences on the start now agree with the code.
  • MongoDB gets a {table_id: 1, shard_id: 1} index on stream_shards: the open-shard read on every write to a streamed table, and the shard reads behind DescribeStream, GetShardIterator and ListStreams, otherwise scan the whole collection. extenddb init creates it; an existing deployment gets it from extenddb migrate --yes, which reports it pending while it is missing.
  • The lifecycle comments and the design document say what a disable stores ({StreamEnabled: false} with the view type, omitted from responses) instead of "clears the specification".

Tests

  • Wire: tests/python/test_stream_disable.py (21), test_stream_disable_fold.py (16), test_stream_disable_validation.py (14), dual-target. Against the live service: 51 of 51 (measured 2026-10-24). Against a server of each backend built from this head: the three files with test_stream_shard_ownership.py, test_streams.py and test_streams_transactions.py give 71 passed, 4 xfailed on each of SQLite, PostgreSQL, MongoDB and Cassandra (the four GetRecords terminal-page rows wait on End GetRecords iterators after closed shard exhaustion #372's rule; every other assertion in them runs). On main at 1d0d8f4, the first file, at 21 rows, fails 14 against a SQLite server (3 pass, 4 xfail).
  • Rust: extenddb-storage --lib 158 (stream_lineage 15, shard_stream 7); extenddb-storage-sqlite --lib 169, 1 ignored (stream_lifecycle 4, stream_tests 10); extenddb-storage-postgres --test stream_disable 5 (lifecycle, scope, the write race, an enable closing an earlier stream left open, a disable applying after an overlapping enable) and --test stream_disable_legacy 2 (the legacy upgrade, the refused disable of a disabled stream), stream_shard_ownership 6, stream_records_page_order 1, stream_shard_retention 1, table_snapshot 81; extenddb-storage-cassandra --test direct_integration stream_disable 2, stream_endings 2, stream_ownership 3, streams 14; extenddb-storage-mongodb live tests 9 (#[ignore] without a server).
  • Controls: each change that alters behaviour has a test that fails on the code before it (the lineage rule, a legacy shard under a newer labelled stream, the shard id maximum, the non-object specification, the refused disable, the write race lock, the Cassandra start, the Cassandra close guard, an enable closing an earlier stream left open), and the late-disable test fails when a close takes every open labelled shard of the table.
  • cargo fmt, clippy 1.97.0 with -D warnings on every target, check on 1.88.0: clean. No dependency change. Every added source file is under 600 lines; crates/engine/src/streams.rs, crates/storage/src/stream_lineage.rs and Cassandra's stream_lifecycle.rs were split by concern to stay under it, and the PostgreSQL and Cassandra stream disable tests are split the same way, the PostgreSQL ones with a shared fixture.
  • CI passed all 35 checks on a42aeb77c after one rerun each of the Test and Cassandra jobs. The first runs failed shape_gate_tests::serve_refuses_a_file_database_with_pending_shape_migrations (SQLite answered "database is locked" to the test's ALTER TABLE) and tests/test_abac.py::TestLeadingZeroCidr::test_leading_zero_octet_accepted_and_matches_decimal (creating the test user's access key answered HTTP 500 on Cassandra); both passed on the rerun.

Merge notes

Re-cut on main at 1d0d8f4 after #543 (877ef39); see the section above. #372 (end GetRecords iterators on a closed shard): its rule is what the four xfail rows wait for.

Related behaviour

Pre-existing on main, measured during this work, filed separately: DescribeStream omits CreationRequestDateTime on every backend; the service keeps a disabled stream of a deleted table describable and listed, every backend removes it with the table; the missing-view-type sentence on UpdateTable (StreamViewType must be specified when StreamEnabled is true here, If stream is being enabled then UpdateViewType is required on the service); the undefined-index-key sentence; AT_SEQUENCE_NUMBER past a closed shard's ending number returns the last record on the service and nothing here. MongoDB's write-race window (a write whose transaction read the shard as open and commits after the close lands past the ending number; 4 of 320 under twelve writers) stays, documented: closing it needs the write transaction to touch the shard document and the stream path to classify the resulting conflict as retryable. No ExtendDB UpdateTable answers TableStatus UPDATING, so the two UpdateTable responses say ACTIVE where the service says UPDATING.

A drained closed shard keeps returning NextShardIterator (#372). Closed streams and their shard rows are never removed while the table exists, so ListStreams and DescribeStream keep them; about 1.5 ms to 3 ms after 50 disable and enable cycles and about 20 ms after 500 on SQLite, about 8 ms after 500 on PostgreSQL (robinnsc's measurements in the review).

A write takes its stream view type from the table's key info, which a server that did not run the stream change keeps cached for up to 30 seconds (auth.cache.soft_ttl_seconds). After a disable and an enable with the other view type, writes through a second PostgreSQL server landed in the new stream with the earlier view type: 140 of 186 writes from NEW_AND_OLD_IMAGES to KEYS_ONLY and 141 of 186 the other way, the last at 29.87 s and 30.17 s after the enable. MongoDB and Cassandra build the record from the same cached key info. On Cassandra, from reading the commit path (transact_stream_statement, #576), a transaction prepared under one stream and committed after a disable and an enable does the same on one server, since that path does not read the shard rows. Before this change such records went into the same stream after its view type changed. Filed as #698.

Closes #482

@LeeroyHannigan

Copy link
Copy Markdown
Collaborator Author

One thing for whoever merges this after #576 (Cassandra transacted writes now record stream records).

The race probe for this branch's disable path measured single-item writers, where the window between resolving the stream capture and writing the record is one request. On the transaction path that window is the whole PREPARE phase: a row read and condition evaluation per op, the marker LWTs, up to five create-race retries with 50 ms polls, the ledger update and the state transition. The Cassandra write path computes the shard from the table id and partition key and does not read the shard rows, so a transacted write acked before a disable lands in the same physical partition as always; whether a consumer can read it depends on its sequence number falling inside a stream's range, and after a disable and a re-enable a number between the closed shard's EndingSequenceNumber and the next stream's start is in neither.

So on the merged tree the count of transacted writes acked before DISABLED and readable in no stream is not bounded by what the single-item probe measured. One run of the race probe with a TransactWriteItems writer mode on the merged tree, on Cassandra and PostgreSQL, would give that count.

@robinnsc robinnsc left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lifecycle itself looks solid. On local SQLite, PostgreSQL, MongoDB and Cassandra, a disable gives DISABLED with the original view type, correct ending numbers, no StreamSpecification and LatestStreamArn kept, and records nothing after. An enable gives a new label and four new shards, newest first in ListStreams, and cross-stream shard ids and forged iterators are refused. Writers racing a disable plus immediate enable (about 40k acked writes, TransactWriteItems included on SQLite and PostgreSQL) lost nothing from before the disable, put nothing in the old stream after it, and every post-enable write was found. Overlapping UpdateTable calls applied exactly one of two identical changes. The three new wire files plus test_streams.py give 62 passed / 4 xfailed on SQLite, PostgreSQL and MongoDB. Backup and restore, delete-and-recreate, TTL deletes, GSI and TransactWriteItems recording all landed in the right stream. The shard id parser round-tripped on 200k generated ids, and the 5 mutants of stream_lineage.rs we tried were all caught.

Merge notes, since the branch conflicts with main now (mostly against #543, 877ef39). The merge notes in the description mention #463/#466/#468/#475, which aren't on main, and not #543:

  • PostgreSQL, SQLite and MongoDB get_stream_records all auto-merge main's body with no conflict marker, ending in main's shard_in_stream, which is table_label == Some(arn_label). Main's handle_get_records also no longer calls validate_shard. After a textual resolution, a closed stream's ARN answers "Stream does not exist" once the table is re-enabled, and a token pairing the new ARN with an old shard id reads the old stream. shard_in_stream on top of parse_shard_id would fix it: a labelled id matches its own label, a legacy id matches when the ARN is the current label. The wire file's _reread catches it, but no Rust test does.
  • Cassandra: main's private stream_table_row and this PR's stream_table_row in stream_lifecycle.rs:387 collide (E0592). Main's get_stream_records takes stream_arn, which breaks read_all in tests/direct/stream_disable.rs:96. Main's records query keys on the client's shard id, which returns nothing for labelled ids. Since physical_shard_id only needs the id, the records read could still start in parallel with the shard-row read, keeping #543's two round trips, with the bounds applied after.
  • From reading only: #576's transacted Cassandra records take the view type at request time and the sequence at commit, so a transaction committing across a disable plus enable could land in the new stream with the old view type.

Other notes:

  • A drained closed shard keeps returning NextShardIterator. That's deferred to #372, but it matters more once disables produce closed shards in normal use (KCL and Lambda finish a shard on the null), so it might be worth landing #372 right after this.
  • Closed streams and their shard rows are never removed while the table exists, so ListStreams and DescribeStream keep them indefinitely. DescribeStream, GetShardIterator and ListStreams group every shard row the table ever had: about 1.5 ms to 3 ms after 50 cycles and about 20 ms after 500 on SQLite, and about 8 ms after 500 on PostgreSQL. Probably fine at realistic cycle counts.
  • test_stream_disable.py has 21 rows rather than 20.


A table has a sequence of streams. `UpdateTable StreamEnabled=false` closes the current one: the stored specification is cleared (DescribeTable omits `StreamSpecification`), `LatestStreamArn` and `LatestStreamLabel` keep naming the closed stream, DescribeStream reports it `DISABLED` with its original view type, and every open shard gets an `EndingSequenceNumber` (its last record's sequence number, or its starting number when it holds none). `StreamEnabled=true` afterwards creates a new stream: a new label later than the previous one, four new shard ids, the requested view type. The closed stream stays describable and listed; ListStreams lists every stream of a table, newest first. Enabling a table whose stream is enabled, and disabling a table with no enabled stream, are refused with `ValidationException`.

Each stream's identity is carried in its shard ids, `shardId-{table_id}-{label_millis:013}{view_type:1}{index:02}`, so no backend stores a stream table. Shard ids from before this scheme (`shardId-{table_id}-{index}`, and the `shardId-{table_name}-{index}` form SQLite and PostgreSQL wrote before v0.1.14) belong to the table's current stream. Writes record only to open shards. The rules shared by the four backends live in `extenddb_storage::stream_lineage`; each backend reads its table row and shard rows and calls them.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No catalog bump also means no gate for an earlier release. On SQLite: base, then head disables and re-enables, then base again. Base wrote 4 of 8 records into the closed legacy shards past their EndingSequenceNumber. On a table head created and disabled, base re-enabled the same ARN and every record landed past the ending numbers. A multi-server deployment upgraded one server at a time would do the same once an upgraded server changes a stream. A bump seems heavy here (forced migrate, and Cassandra has no startup gate), so maybe just a line in the upgrade manual: upgrade every server before changing a table's stream, and roll back by restoring the pre-upgrade database.

// then falls between the old shard's ending number and the new
// shard's starting number: in neither stream, rather than in the new
// one under the old view type.
let start = engine

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With more than one server, I think this start is only right per server. generate() reads only the local wall clock (stream_util.rs:60), while the close takes the last record's sequence from whichever server wrote it. With skew between servers, a writer whose clock is behind writes new-stream records below this start, so they're in no stream. A writer whose clock is ahead can leave the old shard ending above this start, so records in that window show up in both streams. Base had no ranges, so skew only reordered records. Could consider taking max(local, last sequence of each open shard) + 1 at close time, which removes the overlap case, and noting the rest beside the deferred cross-instance item. Design doc line 100 ("starts where the closed one on the same index ended") also doesn't match line 104 or the code. Read from code, not reproduced.

Comment thread crates/engine/src/streams.rs Outdated
SHARD_ID_TOO_SHORT.to_owned(),
));
}
if shard_len > SHARD_ID_MAX_LEN {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check runs before validate_shard, so pre-v0.1.14 shardId-{table_name}-{index:016} ids of tables whose name is over 40 characters are now refused, though the PR keeps them as the current stream's shards. On PostgreSQL, a 49-character table name with name-form shards was readable by base and gets ValidationException on head. botocore only enforces the minimum (a 74-character id passes ParamValidator), so SDK clients do reach this. Might be worth applying the max only when the shard isn't found.

(true, true) => Err(StorageError::Validation(format!(
"Table already has an enabled stream: TableName: {table_name}"
))),
(false, false) => Err(StorageError::Validation(format!(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A stream disabled by an earlier release (stored {StreamEnabled: false}, four open shards) reads as DISABLED, KEYS_ONLY, with no EndingSequenceNumber. This refusal then means only an enable closes it. Reproduced on SQLite: created NEW_AND_OLD_IMAGES, disabled on base, and head answered "Table has no stream to disable". A post-commit close that fails (PostgreSQL update_table.rs:851, then the SDK retry) gets to the same state, and writes keep recording into it until the key-info cache expires. Could the decide step close the current label's open shards when the stored spec is already disabled, while still answering this refusal? A storage test that seeds base's disabled state would pin it. Right now a mutant that skips the close on enable passes the SQLite suite.

let mut cursor = shards_coll
.find(doc! { "table_id": table_id })
.find(
doc! { "table_id": table_id, "ending_sequence_number": mongodb::bson::Bson::Null },

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stream_shards only has the {shard_id: 1} index, so this filter FETCHes the whole collection across all accounts. With about 20k shard docs the explain shows 20,008 examined and 4 returned, and PutItem p50 went from 7.9 ms to 40 ms. Most of that already happens on main with many streamed tables, but this PR adds 4 docs per enable that are never pruned (1,000 cycles on one table doubled PutItem p50 on an unrelated table). A {table_id: 1, shard_id: 1} index in the bootstrapper would cover this filter and its sort, plus describe, validate and list.

WHERE table_id = $1 \
ORDER BY shard_id",
WHERE table_id = $1 AND ending_sequence_number IS NULL \
ORDER BY shard_id FOR SHARE",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every write to a streamed table now share-locks all four open shard rows, so concurrent writers on a table keep extending MultiXacts on the same rows, though the record only goes into one shard. In an isolated pgbench run (this SELECT plus one record insert, 32 clients) it was about 30% fewer transactions/s on one hot table and about 8% across 20 tables. End to end over boto3 it was hidden by the client. Could consider selecting the open set unlocked, then SELECT 1 ... WHERE shard_id = <chosen> AND ending_sequence_number IS NULL FOR KEY SHARE on the chosen row only. That still conflicts with the close's FOR UPDATE and re-checks the predicate, and it got back to about 14k tps against 15k unlocked. The race check held either way: 0 records past an ending number in 40 cycles with 12 writers.

Comment thread crates/engine/src/lib.rs
/// deserialised, where the missing field would otherwise surface as a
/// serialization error; its order against the other request checks is not
/// measured.
pub(crate) fn require_stream_enabled_member(body: &serde_json::Value) -> Result<(), DynamoDbError> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: for a StreamSpecification that isn't an object ("x", 1, []), spec.get("StreamEnabled") is None, so this answers the null-member ValidationException where main answered SerializationException. SDKs can't send those shapes, so it hardly matters, but if spec.is_object() would keep it. Also, the new doc block sits inside deserialize_error's doc comment, so the "Enum validation errors ... SerializationException" lines now document this function, and deserialize_error has none.

/// Apply an `UpdateTable` `StreamSpecification` within the caller's
/// transaction.
///
/// Disabling clears the stored specification, so DescribeTable omits

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: disabling now stores {StreamEnabled: false, StreamViewType} (stored_specification) rather than clearing the spec. The same wording is at storage-postgres/src/stream_lifecycle.rs:219, storage-mongodb/src/stream_lifecycle.rs:196 ("sets the specification to null") and design doc line 96. Similarly, engine/src/backup.rs:494-497 says the member is omitted "as it was before the specification was kept", but main stored {StreamEnabled: false} too, so DescribeBackup of a backup taken after a disable now omits a member main showed.

…a new stream

On UpdateTable StreamEnabled=false the stream is closed: DescribeStream
reports DISABLED with the stream's own view type, every shard carries an
EndingSequenceNumber (its last record's sequence number, or its starting
number when it holds none), DescribeTable omits StreamSpecification, and
LatestStreamArn keeps naming the closed stream. Writes record only to
open shards.

UpdateTable StreamEnabled=true afterwards creates a new stream with a
later label and four new shard ids; the closed stream stays describable
and listed. ListStreams lists every stream of a table, newest first.
Enabling a table whose stream is enabled is refused with "Table already
has an enabled stream: TableName: <name>", and disabling a table with
no enabled stream with "Table has no stream to disable: TableName:
<name>". GetShardIterator on a stream ARN with a shard of the table's
other stream answers "Requested resource not found: Shard does not
exist".

A stream change travels alone: an UpdateTable carrying
StreamSpecification beside BillingMode or ProvisionedThroughput is
refused with "You cannot modify stream status while updating table
IOPS", beside any GlobalSecondaryIndexUpdates entry with "You cannot
create or delete index while changing stream status", beside
DeletionProtectionEnabled with "DeletionProtection modification must be
the only operation in the request", and nothing is applied. A disable
beside a StreamViewType is refused ("If stream is being disabled, then
UpdateViewType must not be specified"). CreateTable with a stream
enabled and no view type is refused ("Table is being created with a
stream enabled, UpdateViewType is required"); a StreamSpecification
without StreamEnabled on CreateTable or UpdateTable gets the model's
null-member sentence; a table created with StreamSpecification
{StreamEnabled: false} stores nothing of it.

ListStreams carries LastEvaluatedStreamArn whenever Limit cut the page,
refuses an ExclusiveStartStreamArn of another table ("ExclusiveStartStreamArn
provided is invalid: <arn>"), Limit under 1 and a start ARN under 37
characters. GetShardIterator refuses a ShardId under 28 or over 65
characters and a SequenceNumber under 21 with the service's constraint
sentences, answers AT_SEQUENCE_NUMBER without a number with
"SequenceNumber cannot be null", and an unknown stream ARN with
"Requested resource not found: Stream does not exist". DescribeStream
refuses a short ExclusiveStartShardId and answers an unknown stream of an
existing table with "Stream: <arn> not found".

Shard ids written before v0.1.14 (shardId-{table_name}-{index}) are the
table's shards: that stream stays describable, listed and readable after
the upgrade, and keeps its view type when disabled (the disable stores
{StreamEnabled: false} with the view type and responses omit the
member). A disable closes only the stream it decided on. On MongoDB and
Cassandra the catalog write of a stream change is conditioned on the
specification and label it read, so of two overlapping changes exactly
one applies and the other is refused; an enable re-reads the row after
inserting its shards and closes them when a disable got in between, so a
DISABLED stream has every shard closed. A write racing a disable
lands before the ending number on PostgreSQL (the open shards are read
FOR SHARE and locked FOR UPDATE by the close) and in no stream on
Cassandra (a new stream starts at a clock value drawn at enable time);
MongoDB keeps a one-transaction window.

Each stream's identity is carried in its shard ids
(shardId-{table_id}-{label_millis}{view_type}{index}), so no backend
needs a schema change; ids written before this scheme belong to the
table's current stream. The rules shared by the four backends live in
extenddb_storage::stream_lineage. On Cassandra, whose write paths
compute a record's partition without reading the shard rows, a stream's
shard is a range of its index's partition.

Re-cut on main after 877ef39 (resolving the shard against its stream
inside the GetRecords read). The not-found sentences use the one set of
helpers in extenddb_storage::util; GetShardIterator with a shard of the
table's other stream answers "Requested resource not found: Shard does
not exist" (util::SHARD_DOES_NOT_EXIST, measured 2026-10-24), replacing
the "Stream: <arn> not found." sentence main kept there. The Cassandra
get_stream_records keeps main's ownership check (the shard row's table
id and util::shard_in_stream) and its two round trips, and reads the
records from the shard's physical partition in parallel with the shard
row, applying the shard's sequence range to the rows it gets back. The
engine keeps main's constraint helper beside the shard id length
constants.

Closes #482
…ames

util::shard_in_stream decided ownership by the table's current label
alone, which was right while a table had one stream. With a stream per
enable, that rule made a closed stream unreadable as soon as the table
was enabled again (its ARN answered "Stream does not exist") and let an
iterator pairing the new ARN with an old shard id read the old stream.

The rule now reads the shard id with stream_lineage::parse_shard_id: a
labelled id belongs to the label it carries, and an id from before
streams were named in shard ids belongs to the table's current stream
while the table has no labelled shard. From a table's first enable under
this release every later stream is labelled, so legacy rows beside a
labelled row are the earlier, closed stream, which is how DescribeStream
and GetShardIterator already group them. All four backends call it from
get_stream_records, so a closed stream stays readable under its own ARN
after the table has a newer one, and a token pairing the newer ARN with
an older stream's shard id, labelled or legacy, is refused.

Whether the table has a labelled shard is read only for a legacy id, so
a labelled id keeps #543's statements: an EXISTS on stream_shards over
the byte-order range of the table's labelled ids on SQLite and
PostgreSQL (stream_lineage::labelled_shard_id_range; COLLATE "C" on
PostgreSQL), a find_one on the {table_id, shard_id} index with the
labelled id pattern on MongoDB, whose legacy ids have 12-digit suffixes,
and the table's shard rows on Cassandra.

Unit tests on the rule (a legacy shard under a table that has a labelled
stream for the current label is not its stream; a labelled shard is
decided without reading the table's shards) and on the labelled range,
and SQLite engine tests reading a re-enabled table: the closed stream
under its own ARN, the new stream under its, the two cross pairings
refused, and the new ARN with a legacy shard of a closed stream refused.
…s they carry

The wire test that pinned a re-enable to the same stream ARN moves to the
new ARN the service answers: the iterators issued before the disable read
the closed stream (the same records a fresh iterator on each of its
shards reads) and none of the keys written after the enable, which are
in the new stream's shards alone.

The Cassandra direct tests read a stream's records by the stream's own
shard id for the partition a key lands in, not by the partition id,
since get_stream_records resolves the shard row the id names. The
ownership tests' relabel moves the table's shard rows to the ids the new
label gives them, as a shard id carries its stream's label, and the
disable test checks that a token pairing one stream's ARN with the other
stream's shard id is refused.
…alizationException

The StreamEnabled member check ran on any non-null StreamSpecification,
so a string, number or array there answered the null-member
ValidationException where main answered SerializationException from
deserialisation. The check now applies to objects only. The doc block
that had landed inside deserialize_error's comment documents the member
check it describes.
GetShardIterator refused a ShardId over 65 characters before looking it
up. The shards a release before v0.1.14 wrote are named after the table
(shardId-{table_name}-{index:016}), so a table whose name is over 40
characters had its stream refused with a ValidationException after the
upgrade, where the previous release read it. botocore enforces only the
minimum, so SDK clients reach this check.

The minimum and the SequenceNumber length are still checked first; the
maximum is applied when the lookup finds no such shard, which is the
answer the service gives since it never has a shard that long. A found
shard over the maximum gets its iterator.
…s left open

A release that stored {StreamEnabled: false} without closing the shards,
or a close that failed after its catalog write, leaves a stream that
DescribeStream reports as DISABLED with four open shards, and writes
keep recording into it until the key info cache expires. A disable on
that table was refused with "Table has no stream to disable" and changed
nothing, so only an enable closed the shards.

The decide step on each backend now closes the current label's open
shards when the stored specification is already disabled, then answers
the same refusal. SQLite closes in the request's transaction and commits
it before answering; PostgreSQL, MongoDB and Cassandra close in the
shard store, as the first disable would have, with no catalog write.

Tests on SQLite and PostgreSQL seed that state: the refusal stands, the
ending numbers are the last record or the starting number, a write after
it records nothing, a second refusal has nothing left to close, and on
SQLite an enable met with the same state closes the open shards too.
Every write to a streamed table locked all four open shard rows FOR
SHARE, so concurrent writers on one table kept extending the same
MultiXacts for a record that goes into one shard. The open set is now
read without a lock, and the chosen row alone is locked FOR KEY SHARE
with its ending_sequence_number re-checked. The close still locks the
open rows FOR UPDATE before taking the ending numbers, which conflicts
with this lock, so a write either commits before the ending number is
taken or finds its row closed and records nothing.

A storage test runs eight writers against a disable and checks that no
record is past its shard's ending number.
stream_shards had the unique shard_id index alone, so the open-shard
read on every write to a streamed table, and the shard reads behind
DescribeStream, GetShardIterator and ListStreams, scanned the whole
collection across all accounts: with about 20k shard documents a write
examined 20,008 to return 4, and PutItem p50 went from 7.9 ms to 40 ms
(measured in review). Each enable now adds four documents that stay, so
the collection grows with every disable and enable cycle.

A {table_id: 1, shard_id: 1} index covers the filter and the sort of
those reads. extenddb init creates it in create_data_db. MongoDB's data
migration, which extenddb migrate --yes runs on every upgrade, creates
it too, so an existing deployment gets it; creating it again is a no-op,
and the migration reports it pending while it is missing. A live test
drops the index, checks it is reported pending, runs the migration and
finds it.
…umbers

A new stream's shards started at a value drawn from the enabling
server's clock alone. With several servers, a record of the closed
stream written by a server whose clock is ahead could sit above that
start, and so read in both the closed stream and the new one.

The start is now one above the larger of this server's next clock value
and the last ending number among the table's shards, taken after the
close. A server whose clock is behind the start can still write a
new-stream record below it, which no stream reads, and on several
servers a raced record from a server whose clock is ahead can still land
above the start; the module and the design document note both cases
beside the deferred cross-instance item. The design document also said
the new shard starts where the closed one ended, which neither the code
nor its later paragraph did.

A close writes a shard's ending number only when the shard has none
(UPDATE ... IF ending_sequence_number = null). An open shard's range has
no upper bound, so a disable whose close read the old shards as open and
took their ending numbers after an overlapping enable had closed them
and started the next stream would otherwise set them to a record of the
new stream from the shared partition, above the new stream's start.

Cassandra direct tests: one moves a closed stream's ending numbers an
hour ahead of the clock and checks that the new stream starts above
them; one runs a close with the old stream's rows as read before the
disable, after the enable and eight writes to the new stream, and checks
that the old shards keep their ending numbers.
…tores

The upgrade manual gets a behaviour change entry: a server on the earlier
release writes records past a closed shard's ending number and re-enables
a stream under its old ARN, so every server is upgraded before any
UpdateTable changes a StreamSpecification, and rollback is a restore of
the pre-upgrade database. No catalog version gate stops the earlier
release, since the change needs no catalog migration. The entry also
says an existing MongoDB deployment gets the new stream_shards index
from extenddb migrate --yes.

The PostgreSQL and MongoDB lifecycle comments and the design document
said a disable clears the specification; it stores {StreamEnabled: false}
with the stream's view type, which DescribeTable omits. DescribeBackup
for a backup taken after a disable keeps showing StreamDescription as
{StreamEnabled: false}, as before the view type was kept, since the
service's answer there is not measured.
…uilder to their own files

Pure moves, no behaviour change: crates/engine/src/streams.rs had grown
to 711 lines with the GetShardIterator bounds, so its unit tests move to
streams_tests.rs; crates/storage/src/stream_lineage.rs had reached 602,
so the ListStreams page builder moves to stream_lineage_list.rs and is
re-exported from the parent module.
…rries

The GetRecords ownership and page-order tests pinned a table's stream
label by updating the catalog row alone. A shard id now carries its
stream's label, so the ownership fixture's relabel moves each shard row
and its records to the ids the new label gives them, and the page-order
test reads under the label the table was created with. The relabel test
also checks that the old shard id under the new ARN is refused.
On PostgreSQL, MongoDB and Cassandra a stream change writes the catalog
row first and changes the shards after it. An enable closed only the
open shards of the label it read. When a disable's close failed after
its catalog write and the client then enabled the stream, and that
enable's close failed too, the table was enabled under the new label
with no shards of its own while the earlier stream's four shards stayed
open: writes recorded into a DISABLED stream, a later disable closed
only the new label's rows, and every later enable closed only the label
it read, so those shards were never closed.

An enable now closes the open shards of every stream of the table older
than the one it creates, legacy rows included
(stream_lineage::open_shard_ids_before). A disable still closes only
the stream it read, so a disable that applies after an overlapping
enable leaves the new stream open. SQLite closes every open shard of
the table in the transaction that writes the catalog row, so it is
unchanged.

PostgreSQL storage tests: an_enable_closes_an_earlier_stream_left_open
seeds an enabled label with no shards beside an earlier stream's open
shards, and the next disable and enable close them;
a_disable_applying_after_an_overlapping_enable_leaves_the_new_stream_open
applies a disable's shard changes for the earlier label after the
enable created the new stream, and the new stream's four shards stay
open.
…iles

tests/stream_disable.rs had grown to 877 lines. The scratch database and
request helpers move to tests/common/stream_disable_fixture.rs, which
both test files include; the lifecycle and race tests stay in
stream_disable.rs, and the two tests of states an earlier release left
(the name-based shards from before v0.1.14, and a disabled stream whose
shards stayed open) move to stream_disable_legacy.rs. The test bodies
are unchanged; the helpers gain `pub`. The PostgreSQL storage step in
the integration workflow runs the new --test stream_disable_legacy.
…hard reads to their own files

Three files this change adds had grown past 600 lines. Pure moves, by
concern:

- stream_lineage.rs (603 lines): labelled_shard_id_range,
  labelled_shard_id_pattern and has_labelled_shard move to
  stream_lineage_labelled.rs, re-exported from stream_lineage.
- storage-cassandra stream_lifecycle.rs (635): the table and shard row
  reads (stream_table_row, stream_table_rows, stream_shard_rows and their
  row helper) move to stream_lifecycle_rows.rs, a child module.
- storage-cassandra tests/direct/stream_disable.rs (634): the two tests
  of a closed stream's ending numbers against the next stream's start
  move to tests/direct/stream_endings.rs; the request helpers they share
  with stream_disable.rs become pub(crate).
@LeeroyHannigan

Copy link
Copy Markdown
Collaborator Author

Thanks for the runs. Re-cut on main after #543, with your points folded in; the commits are below.

  • Merge notes, fix(streams): resolve the shard against its stream inside the GetRecords read #543 (shard_in_stream on parse_shard_id): 6f9089f. A labelled id matches its own label, a legacy id the current label while the table has no labelled shard, so a legacy shard paired with a newer stream's ARN is refused too; unit tests plus SQLite engine tests for the closed stream reading under its ARN and the cross pairings refused. Cassandra: one stream_table_row, main's ownership check kept, the records read runs in parallel with the shard row read with the range applied after, two round trips: 4ca0632. read_all and the ownership tests on labelled ids: 690073d.
  • Upgrade gate (design doc line 98): c818c0a. A behaviour change entry in the upgrade manual: upgrade every server before changing a table's stream, rollback is a restore of the pre-upgrade database. No catalog bump.
  • Cassandra clock skew (stream_lifecycle.rs:115): 7823db8. The start is one above the larger of the local clock value and the last ending number among the table's shards, taken after the close; the behind-clock case is noted beside the cross-instance item, and the design doc's line 100 now says the same as line 104 and the code. Test moves the closed shards' ending numbers an hour ahead and checks the new start is above them. The same commit makes a close write a shard's ending number only when it has none, so a late close cannot move it above the new start.
  • Shard id length (streams.rs:124): cea6f30. The maximum applies when the shard is not found; a found shard over the maximum gets its iterator.
  • Disabled by an earlier release (stream_lineage.rs:102): 7723249. The decide step closes the current label's open shards when the stored specification is already disabled, then answers the refusal; storage tests on SQLite and PostgreSQL seed that state (your mutant that skips the close on enable now fails the SQLite suite). On PostgreSQL, MongoDB and Cassandra an enable also closes the open shards of every earlier stream, so a stream left open after a failed close is closed by the next enable whatever label it read: a8d80ef.
  • MongoDB index (stream_engine.rs:115): b3eae75. {table_id: 1, shard_id: 1} on stream_shards, in the bootstrapper and in the data migration extenddb migrate --yes runs, so existing deployments get it too. No explain check: the crate has no explain pattern; a live test drops the index and checks the migration creates it.
  • PostgreSQL locking (tx_helpers.rs:331): bdba4fb. Unlocked read of the open set, FOR KEY SHARE on the chosen row with the IS NULL re-check; a storage test runs eight writers against a disable and checks for records past an ending number (dropping the re-check puts 8 of 320 past it).
  • lib.rs:164: ff46ea4. is_object(), and the doc block is on the member check.
  • Wording (stream_lifecycle.rs:108 and the others), backup.rs:494: c818c0a. The comments and the design doc say what a disable stores. DescribeBackup keeps showing {StreamEnabled: false} for a backup taken after a disable, as on main; the view type is dropped there since the service's answer is not measured.
  • fix(cassandra): record stream records for transacted writes #576: from reading the commit path, a transaction prepared under one stream and committed after a disable plus enable lands in the new stream with the old view type; the fix is in the commit path, so it is filed separately as Streams: records written after a re-enable can carry the earlier stream's view type #698.
  • NextShardIterator on a drained closed shard and the unpruned closed streams are in the body under Related behaviour. The merge notes name fix(streams): resolve the shard against its stream inside the GetRecords read #543. The first wire file has 21 rows; on main at 1d0d8f4 it fails 14 of them against a SQLite server (3 pass, 4 xfail), and the body says so.

@robinnsc, could you look again at a42aeb7?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Disabling a stream does not close its shards or change its status

2 participants