Repository navigation
fix(streams): disabling a stream closes its shards; enabling makes a new one - #515
LeeroyHannigan wants to merge 15 commits into
Conversation
|
One thing for whoever merges this after #576 (Cassandra transacted writes now record stream records). The race probe for this branch's disable path measured single-item writers, where the window between resolving the stream capture and writing the record is one request. On the transaction path that window is the whole PREPARE phase: a row read and condition evaluation per op, the marker LWTs, up to five create-race retries with 50 ms polls, the ledger update and the state transition. The Cassandra write path computes the shard from the table id and partition key and does not read the shard rows, so a transacted write acked before a disable lands in the same physical partition as always; whether a consumer can read it depends on its sequence number falling inside a stream's range, and after a disable and a re-enable a number between the closed shard's So on the merged tree the count of transacted writes acked before |
a484b48 to
5f22dc4
Compare
robinnsc
left a comment
There was a problem hiding this comment.
The lifecycle itself looks solid. On local SQLite, PostgreSQL, MongoDB and Cassandra, a disable gives DISABLED with the original view type, correct ending numbers, no StreamSpecification and LatestStreamArn kept, and records nothing after. An enable gives a new label and four new shards, newest first in ListStreams, and cross-stream shard ids and forged iterators are refused. Writers racing a disable plus immediate enable (about 40k acked writes, TransactWriteItems included on SQLite and PostgreSQL) lost nothing from before the disable, put nothing in the old stream after it, and every post-enable write was found. Overlapping UpdateTable calls applied exactly one of two identical changes. The three new wire files plus test_streams.py give 62 passed / 4 xfailed on SQLite, PostgreSQL and MongoDB. Backup and restore, delete-and-recreate, TTL deletes, GSI and TransactWriteItems recording all landed in the right stream. The shard id parser round-tripped on 200k generated ids, and the 5 mutants of stream_lineage.rs we tried were all caught.
Merge notes, since the branch conflicts with main now (mostly against #543, 877ef39). The merge notes in the description mention #463/#466/#468/#475, which aren't on main, and not #543:
- PostgreSQL, SQLite and MongoDB
get_stream_recordsall auto-merge main's body with no conflict marker, ending in main'sshard_in_stream, which istable_label == Some(arn_label). Main'shandle_get_recordsalso no longer callsvalidate_shard. After a textual resolution, a closed stream's ARN answers "Stream does not exist" once the table is re-enabled, and a token pairing the new ARN with an old shard id reads the old stream.shard_in_streamon top ofparse_shard_idwould fix it: a labelled id matches its own label, a legacy id matches when the ARN is the current label. The wire file's_rereadcatches it, but no Rust test does. - Cassandra: main's private
stream_table_rowand this PR'sstream_table_rowinstream_lifecycle.rs:387collide (E0592). Main'sget_stream_recordstakesstream_arn, which breaksread_allintests/direct/stream_disable.rs:96. Main's records query keys on the client's shard id, which returns nothing for labelled ids. Sincephysical_shard_idonly needs the id, the records read could still start in parallel with the shard-row read, keeping #543's two round trips, with the bounds applied after. - From reading only: #576's transacted Cassandra records take the view type at request time and the sequence at commit, so a transaction committing across a disable plus enable could land in the new stream with the old view type.
Other notes:
- A drained closed shard keeps returning
NextShardIterator. That's deferred to #372, but it matters more once disables produce closed shards in normal use (KCL and Lambda finish a shard on the null), so it might be worth landing #372 right after this. - Closed streams and their shard rows are never removed while the table exists, so ListStreams and DescribeStream keep them indefinitely. DescribeStream, GetShardIterator and ListStreams group every shard row the table ever had: about 1.5 ms to 3 ms after 50 cycles and about 20 ms after 500 on SQLite, and about 8 ms after 500 on PostgreSQL. Probably fine at realistic cycle counts.
test_stream_disable.pyhas 21 rows rather than 20.
|
|
||
| A table has a sequence of streams. `UpdateTable StreamEnabled=false` closes the current one: the stored specification is cleared (DescribeTable omits `StreamSpecification`), `LatestStreamArn` and `LatestStreamLabel` keep naming the closed stream, DescribeStream reports it `DISABLED` with its original view type, and every open shard gets an `EndingSequenceNumber` (its last record's sequence number, or its starting number when it holds none). `StreamEnabled=true` afterwards creates a new stream: a new label later than the previous one, four new shard ids, the requested view type. The closed stream stays describable and listed; ListStreams lists every stream of a table, newest first. Enabling a table whose stream is enabled, and disabling a table with no enabled stream, are refused with `ValidationException`. | ||
|
|
||
| Each stream's identity is carried in its shard ids, `shardId-{table_id}-{label_millis:013}{view_type:1}{index:02}`, so no backend stores a stream table. Shard ids from before this scheme (`shardId-{table_id}-{index}`, and the `shardId-{table_name}-{index}` form SQLite and PostgreSQL wrote before v0.1.14) belong to the table's current stream. Writes record only to open shards. The rules shared by the four backends live in `extenddb_storage::stream_lineage`; each backend reads its table row and shard rows and calls them. |
There was a problem hiding this comment.
No catalog bump also means no gate for an earlier release. On SQLite: base, then head disables and re-enables, then base again. Base wrote 4 of 8 records into the closed legacy shards past their EndingSequenceNumber. On a table head created and disabled, base re-enabled the same ARN and every record landed past the ending numbers. A multi-server deployment upgraded one server at a time would do the same once an upgraded server changes a stream. A bump seems heavy here (forced migrate, and Cassandra has no startup gate), so maybe just a line in the upgrade manual: upgrade every server before changing a table's stream, and roll back by restoring the pre-upgrade database.
| // then falls between the old shard's ending number and the new | ||
| // shard's starting number: in neither stream, rather than in the new | ||
| // one under the old view type. | ||
| let start = engine |
There was a problem hiding this comment.
With more than one server, I think this start is only right per server. generate() reads only the local wall clock (stream_util.rs:60), while the close takes the last record's sequence from whichever server wrote it. With skew between servers, a writer whose clock is behind writes new-stream records below this start, so they're in no stream. A writer whose clock is ahead can leave the old shard ending above this start, so records in that window show up in both streams. Base had no ranges, so skew only reordered records. Could consider taking max(local, last sequence of each open shard) + 1 at close time, which removes the overlap case, and noting the rest beside the deferred cross-instance item. Design doc line 100 ("starts where the closed one on the same index ended") also doesn't match line 104 or the code. Read from code, not reproduced.
| SHARD_ID_TOO_SHORT.to_owned(), | ||
| )); | ||
| } | ||
| if shard_len > SHARD_ID_MAX_LEN { |
There was a problem hiding this comment.
This check runs before validate_shard, so pre-v0.1.14 shardId-{table_name}-{index:016} ids of tables whose name is over 40 characters are now refused, though the PR keeps them as the current stream's shards. On PostgreSQL, a 49-character table name with name-form shards was readable by base and gets ValidationException on head. botocore only enforces the minimum (a 74-character id passes ParamValidator), so SDK clients do reach this. Might be worth applying the max only when the shard isn't found.
| (true, true) => Err(StorageError::Validation(format!( | ||
| "Table already has an enabled stream: TableName: {table_name}" | ||
| ))), | ||
| (false, false) => Err(StorageError::Validation(format!( |
There was a problem hiding this comment.
A stream disabled by an earlier release (stored {StreamEnabled: false}, four open shards) reads as DISABLED, KEYS_ONLY, with no EndingSequenceNumber. This refusal then means only an enable closes it. Reproduced on SQLite: created NEW_AND_OLD_IMAGES, disabled on base, and head answered "Table has no stream to disable". A post-commit close that fails (PostgreSQL update_table.rs:851, then the SDK retry) gets to the same state, and writes keep recording into it until the key-info cache expires. Could the decide step close the current label's open shards when the stored spec is already disabled, while still answering this refusal? A storage test that seeds base's disabled state would pin it. Right now a mutant that skips the close on enable passes the SQLite suite.
| let mut cursor = shards_coll | ||
| .find(doc! { "table_id": table_id }) | ||
| .find( | ||
| doc! { "table_id": table_id, "ending_sequence_number": mongodb::bson::Bson::Null }, |
There was a problem hiding this comment.
stream_shards only has the {shard_id: 1} index, so this filter FETCHes the whole collection across all accounts. With about 20k shard docs the explain shows 20,008 examined and 4 returned, and PutItem p50 went from 7.9 ms to 40 ms. Most of that already happens on main with many streamed tables, but this PR adds 4 docs per enable that are never pruned (1,000 cycles on one table doubled PutItem p50 on an unrelated table). A {table_id: 1, shard_id: 1} index in the bootstrapper would cover this filter and its sort, plus describe, validate and list.
| WHERE table_id = $1 \ | ||
| ORDER BY shard_id", | ||
| WHERE table_id = $1 AND ending_sequence_number IS NULL \ | ||
| ORDER BY shard_id FOR SHARE", |
There was a problem hiding this comment.
Every write to a streamed table now share-locks all four open shard rows, so concurrent writers on a table keep extending MultiXacts on the same rows, though the record only goes into one shard. In an isolated pgbench run (this SELECT plus one record insert, 32 clients) it was about 30% fewer transactions/s on one hot table and about 8% across 20 tables. End to end over boto3 it was hidden by the client. Could consider selecting the open set unlocked, then SELECT 1 ... WHERE shard_id = <chosen> AND ending_sequence_number IS NULL FOR KEY SHARE on the chosen row only. That still conflicts with the close's FOR UPDATE and re-checks the predicate, and it got back to about 14k tps against 15k unlocked. The race check held either way: 0 records past an ending number in 40 cycles with 12 writers.
| /// deserialised, where the missing field would otherwise surface as a | ||
| /// serialization error; its order against the other request checks is not | ||
| /// measured. | ||
| pub(crate) fn require_stream_enabled_member(body: &serde_json::Value) -> Result<(), DynamoDbError> { |
There was a problem hiding this comment.
nit: for a StreamSpecification that isn't an object ("x", 1, []), spec.get("StreamEnabled") is None, so this answers the null-member ValidationException where main answered SerializationException. SDKs can't send those shapes, so it hardly matters, but if spec.is_object() would keep it. Also, the new doc block sits inside deserialize_error's doc comment, so the "Enum validation errors ... SerializationException" lines now document this function, and deserialize_error has none.
| /// Apply an `UpdateTable` `StreamSpecification` within the caller's | ||
| /// transaction. | ||
| /// | ||
| /// Disabling clears the stored specification, so DescribeTable omits |
There was a problem hiding this comment.
nit: disabling now stores {StreamEnabled: false, StreamViewType} (stored_specification) rather than clearing the spec. The same wording is at storage-postgres/src/stream_lifecycle.rs:219, storage-mongodb/src/stream_lifecycle.rs:196 ("sets the specification to null") and design doc line 96. Similarly, engine/src/backup.rs:494-497 says the member is omitted "as it was before the specification was kept", but main stored {StreamEnabled: false} too, so DescribeBackup of a backup taken after a disable now omits a member main showed.
…a new stream
On UpdateTable StreamEnabled=false the stream is closed: DescribeStream
reports DISABLED with the stream's own view type, every shard carries an
EndingSequenceNumber (its last record's sequence number, or its starting
number when it holds none), DescribeTable omits StreamSpecification, and
LatestStreamArn keeps naming the closed stream. Writes record only to
open shards.
UpdateTable StreamEnabled=true afterwards creates a new stream with a
later label and four new shard ids; the closed stream stays describable
and listed. ListStreams lists every stream of a table, newest first.
Enabling a table whose stream is enabled is refused with "Table already
has an enabled stream: TableName: <name>", and disabling a table with
no enabled stream with "Table has no stream to disable: TableName:
<name>". GetShardIterator on a stream ARN with a shard of the table's
other stream answers "Requested resource not found: Shard does not
exist".
A stream change travels alone: an UpdateTable carrying
StreamSpecification beside BillingMode or ProvisionedThroughput is
refused with "You cannot modify stream status while updating table
IOPS", beside any GlobalSecondaryIndexUpdates entry with "You cannot
create or delete index while changing stream status", beside
DeletionProtectionEnabled with "DeletionProtection modification must be
the only operation in the request", and nothing is applied. A disable
beside a StreamViewType is refused ("If stream is being disabled, then
UpdateViewType must not be specified"). CreateTable with a stream
enabled and no view type is refused ("Table is being created with a
stream enabled, UpdateViewType is required"); a StreamSpecification
without StreamEnabled on CreateTable or UpdateTable gets the model's
null-member sentence; a table created with StreamSpecification
{StreamEnabled: false} stores nothing of it.
ListStreams carries LastEvaluatedStreamArn whenever Limit cut the page,
refuses an ExclusiveStartStreamArn of another table ("ExclusiveStartStreamArn
provided is invalid: <arn>"), Limit under 1 and a start ARN under 37
characters. GetShardIterator refuses a ShardId under 28 or over 65
characters and a SequenceNumber under 21 with the service's constraint
sentences, answers AT_SEQUENCE_NUMBER without a number with
"SequenceNumber cannot be null", and an unknown stream ARN with
"Requested resource not found: Stream does not exist". DescribeStream
refuses a short ExclusiveStartShardId and answers an unknown stream of an
existing table with "Stream: <arn> not found".
Shard ids written before v0.1.14 (shardId-{table_name}-{index}) are the
table's shards: that stream stays describable, listed and readable after
the upgrade, and keeps its view type when disabled (the disable stores
{StreamEnabled: false} with the view type and responses omit the
member). A disable closes only the stream it decided on. On MongoDB and
Cassandra the catalog write of a stream change is conditioned on the
specification and label it read, so of two overlapping changes exactly
one applies and the other is refused; an enable re-reads the row after
inserting its shards and closes them when a disable got in between, so a
DISABLED stream has every shard closed. A write racing a disable
lands before the ending number on PostgreSQL (the open shards are read
FOR SHARE and locked FOR UPDATE by the close) and in no stream on
Cassandra (a new stream starts at a clock value drawn at enable time);
MongoDB keeps a one-transaction window.
Each stream's identity is carried in its shard ids
(shardId-{table_id}-{label_millis}{view_type}{index}), so no backend
needs a schema change; ids written before this scheme belong to the
table's current stream. The rules shared by the four backends live in
extenddb_storage::stream_lineage. On Cassandra, whose write paths
compute a record's partition without reading the shard rows, a stream's
shard is a range of its index's partition.
Re-cut on main after 877ef39 (resolving the shard against its stream
inside the GetRecords read). The not-found sentences use the one set of
helpers in extenddb_storage::util; GetShardIterator with a shard of the
table's other stream answers "Requested resource not found: Shard does
not exist" (util::SHARD_DOES_NOT_EXIST, measured 2026-10-24), replacing
the "Stream: <arn> not found." sentence main kept there. The Cassandra
get_stream_records keeps main's ownership check (the shard row's table
id and util::shard_in_stream) and its two round trips, and reads the
records from the shard's physical partition in parallel with the shard
row, applying the shard's sequence range to the rows it gets back. The
engine keeps main's constraint helper beside the shard id length
constants.
Closes #482
…ames util::shard_in_stream decided ownership by the table's current label alone, which was right while a table had one stream. With a stream per enable, that rule made a closed stream unreadable as soon as the table was enabled again (its ARN answered "Stream does not exist") and let an iterator pairing the new ARN with an old shard id read the old stream. The rule now reads the shard id with stream_lineage::parse_shard_id: a labelled id belongs to the label it carries, and an id from before streams were named in shard ids belongs to the table's current stream while the table has no labelled shard. From a table's first enable under this release every later stream is labelled, so legacy rows beside a labelled row are the earlier, closed stream, which is how DescribeStream and GetShardIterator already group them. All four backends call it from get_stream_records, so a closed stream stays readable under its own ARN after the table has a newer one, and a token pairing the newer ARN with an older stream's shard id, labelled or legacy, is refused. Whether the table has a labelled shard is read only for a legacy id, so a labelled id keeps #543's statements: an EXISTS on stream_shards over the byte-order range of the table's labelled ids on SQLite and PostgreSQL (stream_lineage::labelled_shard_id_range; COLLATE "C" on PostgreSQL), a find_one on the {table_id, shard_id} index with the labelled id pattern on MongoDB, whose legacy ids have 12-digit suffixes, and the table's shard rows on Cassandra. Unit tests on the rule (a legacy shard under a table that has a labelled stream for the current label is not its stream; a labelled shard is decided without reading the table's shards) and on the labelled range, and SQLite engine tests reading a re-enabled table: the closed stream under its own ARN, the new stream under its, the two cross pairings refused, and the new ARN with a legacy shard of a closed stream refused.
…s they carry The wire test that pinned a re-enable to the same stream ARN moves to the new ARN the service answers: the iterators issued before the disable read the closed stream (the same records a fresh iterator on each of its shards reads) and none of the keys written after the enable, which are in the new stream's shards alone. The Cassandra direct tests read a stream's records by the stream's own shard id for the partition a key lands in, not by the partition id, since get_stream_records resolves the shard row the id names. The ownership tests' relabel moves the table's shard rows to the ids the new label gives them, as a shard id carries its stream's label, and the disable test checks that a token pairing one stream's ARN with the other stream's shard id is refused.
…alizationException The StreamEnabled member check ran on any non-null StreamSpecification, so a string, number or array there answered the null-member ValidationException where main answered SerializationException from deserialisation. The check now applies to objects only. The doc block that had landed inside deserialize_error's comment documents the member check it describes.
GetShardIterator refused a ShardId over 65 characters before looking it
up. The shards a release before v0.1.14 wrote are named after the table
(shardId-{table_name}-{index:016}), so a table whose name is over 40
characters had its stream refused with a ValidationException after the
upgrade, where the previous release read it. botocore enforces only the
minimum, so SDK clients reach this check.
The minimum and the SequenceNumber length are still checked first; the
maximum is applied when the lookup finds no such shard, which is the
answer the service gives since it never has a shard that long. A found
shard over the maximum gets its iterator.
…s left open
A release that stored {StreamEnabled: false} without closing the shards,
or a close that failed after its catalog write, leaves a stream that
DescribeStream reports as DISABLED with four open shards, and writes
keep recording into it until the key info cache expires. A disable on
that table was refused with "Table has no stream to disable" and changed
nothing, so only an enable closed the shards.
The decide step on each backend now closes the current label's open
shards when the stored specification is already disabled, then answers
the same refusal. SQLite closes in the request's transaction and commits
it before answering; PostgreSQL, MongoDB and Cassandra close in the
shard store, as the first disable would have, with no catalog write.
Tests on SQLite and PostgreSQL seed that state: the refusal stands, the
ending numbers are the last record or the starting number, a write after
it records nothing, a second refusal has nothing left to close, and on
SQLite an enable met with the same state closes the open shards too.
Every write to a streamed table locked all four open shard rows FOR SHARE, so concurrent writers on one table kept extending the same MultiXacts for a record that goes into one shard. The open set is now read without a lock, and the chosen row alone is locked FOR KEY SHARE with its ending_sequence_number re-checked. The close still locks the open rows FOR UPDATE before taking the ending numbers, which conflicts with this lock, so a write either commits before the ending number is taken or finds its row closed and records nothing. A storage test runs eight writers against a disable and checks that no record is past its shard's ending number.
stream_shards had the unique shard_id index alone, so the open-shard
read on every write to a streamed table, and the shard reads behind
DescribeStream, GetShardIterator and ListStreams, scanned the whole
collection across all accounts: with about 20k shard documents a write
examined 20,008 to return 4, and PutItem p50 went from 7.9 ms to 40 ms
(measured in review). Each enable now adds four documents that stay, so
the collection grows with every disable and enable cycle.
A {table_id: 1, shard_id: 1} index covers the filter and the sort of
those reads. extenddb init creates it in create_data_db. MongoDB's data
migration, which extenddb migrate --yes runs on every upgrade, creates
it too, so an existing deployment gets it; creating it again is a no-op,
and the migration reports it pending while it is missing. A live test
drops the index, checks it is reported pending, runs the migration and
finds it.
…umbers A new stream's shards started at a value drawn from the enabling server's clock alone. With several servers, a record of the closed stream written by a server whose clock is ahead could sit above that start, and so read in both the closed stream and the new one. The start is now one above the larger of this server's next clock value and the last ending number among the table's shards, taken after the close. A server whose clock is behind the start can still write a new-stream record below it, which no stream reads, and on several servers a raced record from a server whose clock is ahead can still land above the start; the module and the design document note both cases beside the deferred cross-instance item. The design document also said the new shard starts where the closed one ended, which neither the code nor its later paragraph did. A close writes a shard's ending number only when the shard has none (UPDATE ... IF ending_sequence_number = null). An open shard's range has no upper bound, so a disable whose close read the old shards as open and took their ending numbers after an overlapping enable had closed them and started the next stream would otherwise set them to a record of the new stream from the shared partition, above the new stream's start. Cassandra direct tests: one moves a closed stream's ending numbers an hour ahead of the clock and checks that the new stream starts above them; one runs a close with the old stream's rows as read before the disable, after the enable and eight writes to the new stream, and checks that the old shards keep their ending numbers.
…tores
The upgrade manual gets a behaviour change entry: a server on the earlier
release writes records past a closed shard's ending number and re-enables
a stream under its old ARN, so every server is upgraded before any
UpdateTable changes a StreamSpecification, and rollback is a restore of
the pre-upgrade database. No catalog version gate stops the earlier
release, since the change needs no catalog migration. The entry also
says an existing MongoDB deployment gets the new stream_shards index
from extenddb migrate --yes.
The PostgreSQL and MongoDB lifecycle comments and the design document
said a disable clears the specification; it stores {StreamEnabled: false}
with the stream's view type, which DescribeTable omits. DescribeBackup
for a backup taken after a disable keeps showing StreamDescription as
{StreamEnabled: false}, as before the view type was kept, since the
service's answer there is not measured.
…uilder to their own files Pure moves, no behaviour change: crates/engine/src/streams.rs had grown to 711 lines with the GetShardIterator bounds, so its unit tests move to streams_tests.rs; crates/storage/src/stream_lineage.rs had reached 602, so the ListStreams page builder moves to stream_lineage_list.rs and is re-exported from the parent module.
…rries The GetRecords ownership and page-order tests pinned a table's stream label by updating the catalog row alone. A shard id now carries its stream's label, so the ownership fixture's relabel moves each shard row and its records to the ids the new label gives them, and the page-order test reads under the label the table was created with. The relabel test also checks that the old shard id under the new ARN is refused.
On PostgreSQL, MongoDB and Cassandra a stream change writes the catalog row first and changes the shards after it. An enable closed only the open shards of the label it read. When a disable's close failed after its catalog write and the client then enabled the stream, and that enable's close failed too, the table was enabled under the new label with no shards of its own while the earlier stream's four shards stayed open: writes recorded into a DISABLED stream, a later disable closed only the new label's rows, and every later enable closed only the label it read, so those shards were never closed. An enable now closes the open shards of every stream of the table older than the one it creates, legacy rows included (stream_lineage::open_shard_ids_before). A disable still closes only the stream it read, so a disable that applies after an overlapping enable leaves the new stream open. SQLite closes every open shard of the table in the transaction that writes the catalog row, so it is unchanged. PostgreSQL storage tests: an_enable_closes_an_earlier_stream_left_open seeds an enabled label with no shards beside an earlier stream's open shards, and the next disable and enable close them; a_disable_applying_after_an_overlapping_enable_leaves_the_new_stream_open applies a disable's shard changes for the earlier label after the enable created the new stream, and the new stream's four shards stay open.
…iles tests/stream_disable.rs had grown to 877 lines. The scratch database and request helpers move to tests/common/stream_disable_fixture.rs, which both test files include; the lifecycle and race tests stay in stream_disable.rs, and the two tests of states an earlier release left (the name-based shards from before v0.1.14, and a disabled stream whose shards stayed open) move to stream_disable_legacy.rs. The test bodies are unchanged; the helpers gain `pub`. The PostgreSQL storage step in the integration workflow runs the new --test stream_disable_legacy.
…hard reads to their own files Three files this change adds had grown past 600 lines. Pure moves, by concern: - stream_lineage.rs (603 lines): labelled_shard_id_range, labelled_shard_id_pattern and has_labelled_shard move to stream_lineage_labelled.rs, re-exported from stream_lineage. - storage-cassandra stream_lifecycle.rs (635): the table and shard row reads (stream_table_row, stream_table_rows, stream_shard_rows and their row helper) move to stream_lifecycle_rows.rs, a child module. - storage-cassandra tests/direct/stream_disable.rs (634): the two tests of a closed stream's ending numbers against the next stream's start move to tests/direct/stream_endings.rs; the request helpers they share with stream_disable.rs become pub(crate).
5f22dc4 to
a42aeb7
Compare
|
Thanks for the runs. Re-cut on main after #543, with your points folded in; the commits are below.
|
Summary
UpdateTable StreamEnabled=falseleft the streamENABLEDwith every shard open on all four backends, and a later enable reused the same stream ARN, label and shard ids (#482). Measured on the service: a disable closes the stream (DISABLED, every shard with anEndingSequenceNumber,DescribeTablewithoutStreamSpecification,LatestStreamArnstill naming it), an enable afterwards creates a new stream with a later label and four new shards, both streams stay listed, and enabling an enabled table or disabling one with no stream is refused.A table now has a sequence of streams. Each stream's identity is carried in its shard ids (
shardId-{table_id}-{label_millis}{view_type}{index}), so no backend needs a schema change, a migration or a catalog version bump; ids written before this scheme, including theshardId-{table_name}-{index}form from before v0.1.14, belong to the table's current stream until its first enable under this release. The rules shared by the four backends (grouping shards into streams, status, paging, the refusals) live inextenddb_storage::stream_lineage; each backend has astream_lifecycle.rsthat reads its rows and calls them.What a client sees
Every sentence below was measured on the service (two repeats each) and is answered byte for byte on all four backends:
DescribeStreamreportsDISABLEDwith the stream's own view type; each shard'sEndingSequenceNumberis its last record's sequence number, or its starting number when empty;DescribeTableand theUpdateTableresponse omitStreamSpecification;LatestStreamArnkeeps the closed stream. Writes record only to open shards.ListStreamslists both, newest first; a shard id of one stream on the other's ARN isRequested resource not found: Shard does not exist.Table already has an enabled stream: TableName: <name>;Table has no stream to disable: TableName: <name>; a stream change besideBillingModeorProvisionedThroughput(You cannot modify stream status while updating table IOPS), besideGlobalSecondaryIndexUpdates(You cannot create or delete index while changing stream status), besideDeletionProtectionEnabled(DeletionProtection modification must be the only operation in the request), nothing applied; a disable carrying a view type (If stream is being disabled, then UpdateViewType must not be specified);CreateTablewith a stream enabled and no view type (Table is being created with a stream enabled, UpdateViewType is required); aStreamSpecificationobject withoutStreamEnabled(the model's null-member sentence); a table created with{StreamEnabled: false}stores nothing of it.ListStreamscarriesLastEvaluatedStreamArnwheneverLimitcut the page, refuses a start ARN of another table,Limitunder 1 and a start ARN under 37 characters;GetShardIteratorrefuses aShardIdunder 28 or over 65 characters and aSequenceNumberunder 21, answersAT_SEQUENCE_NUMBERwithout a number withSequenceNumber cannot be nulland an unknown stream ARN withStream does not exist;DescribeStreamrefuses a shortExclusiveStartShardIdand answers an unknown stream of an existing table withStream: <arn> not found.Consistency under concurrency
DISABLEDlands before the ending number. On PostgreSQL the write path reads the open shards without a lock, locks the one shard its record goes toFOR KEY SHAREwhile re-checking that it is still open, and the close locks the open rowsFOR UPDATEbefore taking the ending numbers, so a raced write lands before the ending number or records nothing; on Cassandra a new stream starts one above the larger of the enabling server's next clock value and the last ending number among the table's shards, so on one server a raced record of the closed stream falls in neither stream rather than in the new one; with several servers, a record written by a server whose clock is ahead of the enabling server's can land in the new stream (part of the cross-instance consistency item in the design document). Measured with twelve writer threads under the earlier lock shape: 0 records past the ending number and 0 in the next stream on both (before: 5 of 837 and 1); a storage test now runs eight writers against a disable on PostgreSQL and checks that no record is past its shard's ending number. MongoDB keeps a one-transaction window (see below).DISABLEDstream has every shard closed (0 of 60 rounds on PostgreSQL and 0 of 30 on MongoDB; before: 11 of 30 and 2 of 30).IF ending_sequence_number = null), so a disable whose close runs after an overlapping enable started the next stream cannot raise the closed shard's ending number above the new stream's start.{StreamEnabled: false}, four open shards), or left so by a close that failed after its catalog write, keeps recording writes. A disable on that table is refused as before and closes the current label's open shards first; an enable closes them too. Storage tests on SQLite and PostgreSQL seed that state. On PostgreSQL, MongoDB and Cassandra an enable closes the open shards of every earlier stream of the table, legacy rows included, so a stream left open after two failed closes in a row (a disable's, then the next enable's) is closed by the enable after that; a PostgreSQL storage test seeds that state. SQLite closes every open shard of the table in the transaction that writes the catalog row.Compatibility
A stream created by an earlier release keeps working:
DescribeStream,ListStreams,GetShardIterator,GetRecordsand writes before any change; a disable closes it and keeps its view type; an enable replaces it with a labelled stream. Pinned by a PostgreSQL storage test that seeds the pre-v0.1.14 rows. One limitation: such a stream's label lived only in the table row, so after a disable and a re-enable its ARN no longer resolves:DescribeStreamanswersStream: <arn> not found,ListStreamsomits it, and an iterator still reading it answersStream does not exist, so drain such a stream before enabling it again (a disable alone keeps it describable). Recording it would need a per-table stream history column and a migration on three backends; documented in the differences table.There is no catalog version gate, so a server still on the earlier release writes records past a closed shard's ending number and re-enables a stream under its old ARN. The upgrade manual says to upgrade every server before changing a table's stream, and that rollback is a restore of the pre-upgrade database.
DescribeBackupfor a backup taken after a disable keeps showingStreamDescription: {StreamEnabled: false}, as before; the view type the backends now keep for the closed stream is not added there, since the service's answer for that backup is not measured.Re-cut onto #543
#543 (877ef39) resolves the shard against its stream inside the
GetRecordsread, with the table's one label deciding ownership. This branch is re-cut on top of it:extenddb_storage::util.GetShardIteratorwith a shard of the table's other stream, or a well-formed shard id the stream never had, answersRequested resource not found: Shard does not exist(both measured 2026-10-24); main answeredStream: <arn> not found.with a period there.util::shard_in_streamreads the shard id: a labelled id belongs to the label it carries, and a legacy id to the table's current label while the table has no labelled shard. From a table's first enable under this release every later stream is labelled, which makes legacy rows beside a labelled row the earlier, closed stream. So a closed stream stays readable under its own ARN after the table has a newer one, and a token pairing the newer ARN with an older stream's shard id, labelled or legacy, is refused, asGetShardIteratorrefuses the same pairing. The labelled-shard check runs only for a legacy id: anEXISTSonstream_shardson SQLite and PostgreSQL (byte order,COLLATE "C"on PostgreSQL), afind_oneon the{table_id, shard_id}index on MongoDB, the table's shard rows on Cassandra; a labelled id keeps fix(streams): resolve the shard against its stream inside the GetRecords read #543's statements. Unit tests on the rule, SQLite engine tests on a re-enabled table and on a table whose legacy stream was closed by a disable and replaced by an enable; the wire fixture's reread after the re-enable covers the first on every backend.get_stream_recordskeeps main's ownership check (the shard row's table id andshard_in_stream) and its two round trips, reading the records from the shard's physical partition in parallel with the shard row and applying the shard's sequence range to the rows it gets back. Onestream_table_row.constrainthelper beside the shard id length constants.Review changes
GetShardIteratorapplies the 65-characterShardIdmaximum only when the shard is not found, so theshardId-{table_name}-{index:016}shards of a table named with more than 40 characters stay readable.StreamSpecificationthat is not an object keepsSerializationExceptionfrom deserialisation; the null-member check applies to objects.FOR KEY SHAREinstead of all fourFOR SHARE(above).{table_id: 1, shard_id: 1}index onstream_shards: the open-shard read on every write to a streamed table, and the shard reads behindDescribeStream,GetShardIteratorandListStreams, otherwise scan the whole collection.extenddb initcreates it; an existing deployment gets it fromextenddb migrate --yes, which reports it pending while it is missing.{StreamEnabled: false}with the view type, omitted from responses) instead of "clears the specification".Tests
tests/python/test_stream_disable.py(21),test_stream_disable_fold.py(16),test_stream_disable_validation.py(14), dual-target. Against the live service: 51 of 51 (measured 2026-10-24). Against a server of each backend built from this head: the three files withtest_stream_shard_ownership.py,test_streams.pyandtest_streams_transactions.pygive 71 passed, 4 xfailed on each of SQLite, PostgreSQL, MongoDB and Cassandra (the fourGetRecordsterminal-page rows wait on End GetRecords iterators after closed shard exhaustion #372's rule; every other assertion in them runs). Onmainat 1d0d8f4, the first file, at 21 rows, fails 14 against a SQLite server (3 pass, 4 xfail).extenddb-storage --lib158 (stream_lineage15,shard_stream7);extenddb-storage-sqlite --lib169, 1 ignored (stream_lifecycle4,stream_tests10);extenddb-storage-postgres --test stream_disable5 (lifecycle, scope, the write race, an enable closing an earlier stream left open, a disable applying after an overlapping enable) and--test stream_disable_legacy2 (the legacy upgrade, the refused disable of a disabled stream),stream_shard_ownership6,stream_records_page_order1,stream_shard_retention1,table_snapshot81;extenddb-storage-cassandra --test direct_integrationstream_disable2,stream_endings2,stream_ownership3,streams14;extenddb-storage-mongodblive tests 9 (#[ignore]without a server).cargo fmt,clippy1.97.0 with-D warningson every target,checkon 1.88.0: clean. No dependency change. Every added source file is under 600 lines;crates/engine/src/streams.rs,crates/storage/src/stream_lineage.rsand Cassandra'sstream_lifecycle.rswere split by concern to stay under it, and the PostgreSQL and Cassandra stream disable tests are split the same way, the PostgreSQL ones with a shared fixture.a42aeb77cafter one rerun each of the Test and Cassandra jobs. The first runs failedshape_gate_tests::serve_refuses_a_file_database_with_pending_shape_migrations(SQLite answered "database is locked" to the test'sALTER TABLE) andtests/test_abac.py::TestLeadingZeroCidr::test_leading_zero_octet_accepted_and_matches_decimal(creating the test user's access key answered HTTP 500 on Cassandra); both passed on the rerun.Merge notes
Re-cut on
mainat 1d0d8f4 after #543 (877ef39); see the section above. #372 (endGetRecordsiterators on a closed shard): its rule is what the four xfail rows wait for.Related behaviour
Pre-existing on
main, measured during this work, filed separately:DescribeStreamomitsCreationRequestDateTimeon every backend; the service keeps a disabled stream of a deleted table describable and listed, every backend removes it with the table; the missing-view-type sentence onUpdateTable(StreamViewType must be specified when StreamEnabled is truehere,If stream is being enabled then UpdateViewType is requiredon the service); the undefined-index-key sentence;AT_SEQUENCE_NUMBERpast a closed shard's ending number returns the last record on the service and nothing here. MongoDB's write-race window (a write whose transaction read the shard as open and commits after the close lands past the ending number; 4 of 320 under twelve writers) stays, documented: closing it needs the write transaction to touch the shard document and the stream path to classify the resulting conflict as retryable. No ExtendDBUpdateTableanswersTableStatus UPDATING, so the twoUpdateTableresponses sayACTIVEwhere the service saysUPDATING.A drained closed shard keeps returning
NextShardIterator(#372). Closed streams and their shard rows are never removed while the table exists, soListStreamsandDescribeStreamkeep them; about 1.5 ms to 3 ms after 50 disable and enable cycles and about 20 ms after 500 on SQLite, about 8 ms after 500 on PostgreSQL (robinnsc's measurements in the review).A write takes its stream view type from the table's key info, which a server that did not run the stream change keeps cached for up to 30 seconds (
auth.cache.soft_ttl_seconds). After a disable and an enable with the other view type, writes through a second PostgreSQL server landed in the new stream with the earlier view type: 140 of 186 writes fromNEW_AND_OLD_IMAGEStoKEYS_ONLYand 141 of 186 the other way, the last at 29.87 s and 30.17 s after the enable. MongoDB and Cassandra build the record from the same cached key info. On Cassandra, from reading the commit path (transact_stream_statement, #576), a transaction prepared under one stream and committed after a disable and an enable does the same on one server, since that path does not read the shard rows. Before this change such records went into the same stream after its view type changed. Filed as #698.Closes #482