Repository navigation

[Bridge] Served the stream service's Nexus calls in Core. - #63

Closed

moetemp wants to merge 205 commits into

moe/ford/v2/f4-worker-storefrom

moe/bridge/cn4-nexus-stream-calls

Closed

[Bridge] Served the stream service's Nexus calls in Core.#63
moetemp wants to merge 205 commits into
moe/ford/v2/f4-worker-storefrom
moe/bridge/cn4-nexus-stream-calls

Commits on Oct 9, 2026
- Added the output stream commit and marker protos.
  
  moetemp
  and
  mfateev
  committed
  5a4443d
- Recorded committed stream output in a marker.
  
  moetemp
  and
  mfateev
  committed
  06abd99
- Replayed recorded stream output instead of staging it again.
  
  moetemp
  and
  mfateev
  committed
  4fd166e
Commits on Oct 10, 2026
- Removed the speculative-task discard of output markers.
  moetemp
  committed
  b4b39b3
- Dropped a redundant command prepare and moved a stray comment.
  moetemp
  committed
  1ad12c3
- Merged the c1b review fixes into c1c.
  moetemp
  committed
  a9d57f4
- Ignored the run id when checking a replayed output commit.
  moetemp
  committed
  f2275a5
- Failed a replayed task that commits less output than History.
  moetemp
  committed
  f8b8baf
- Tested several output commits around local activities in one task.
  moetemp
  committed
  dcfd350
- Corrected where replayed manifests arrive and a rustdoc dash.
  moetemp
  committed
  982d076
- Formatted the nondeterminism matcher in the stream tests.
  moetemp
  committed
  7774bbf
- Made the leftover-manifest tests fail fast instead of hanging.
  moetemp
  committed
  ca70524
- Tested output commits around a heartbeated local activity.
  moetemp
  committed
  3961870
- Named the mock failure matcher type for clippy.
  moetemp
  committed
  75b0f70
- Said what the output manifest's topic fields cover.
  moetemp
  committed
  519faa2
- Merged the c1a review fixes into c1b-output-marker.
  moetemp
  committed
  162e7ab
- Split the output manifest refusal by field and noted rollback.
  moetemp
  committed
  2955a1d
- Merged the c1b review fixes into c1c-output-replay.
  moetemp
  committed
  d75a696
- Left the provider id out of the replayed output compare.
  moetemp
  committed
  624d66b
- Failed replay on an output marker whose details don't decode.
  moetemp
  committed
  11b6071
Commits on Oct 11, 2026
- Added the stream envelope and store service protos.
  moetemp
  committed
  db43a49
- Added the version 2 stream fingerprint.
  moetemp
  committed
  4b153ee
- Merged the version 2 fingerprint into c1c-output-replay.
  moetemp
  committed
  637fe04
- Merged the version 2 fingerprint into c2a-stream-protos.
  moetemp
  committed
  7b4e580
- Added the streams crate with the store trait, cursors and records.
  moetemp
  committed
  65a4c7f
- Sent a Workflow's stream records in the commit instead of a manifest.
  moetemp
  committed
  32cb36d
- Added supersession and stale marks to the stream reader.
  moetemp
  committed
  9561640
- Built the output manifest in Core from the committed records.
  moetemp
  committed
  e75bd83
- Added the memory stream store and the store conformance suite.
  moetemp
  committed
  eabc5d3
- Checked replayed records against the recorded manifest in Core.
  moetemp
  committed
  eafa5d8
- Merged the record-carrying commit into c2a-stream-protos.
  moetemp
  committed
  cb63d94
- Merged the record-carrying commit into c2b-streams-crate.
  moetemp
  committed
  6d314cd
- Added the Redis stream store behind the redis feature.
  moetemp
  committed
  42c2351
- Checked cursors against the Python vectors and dropped an unused one.
  moetemp
  committed
  4b2c867
- Ported the Redis retention tests to the Core store.
  moetemp
  committed
  e0e8168
- Merged the record-carrying commit into c2c-reader-supersession.
  moetemp
  committed
  e9ecd90
- Merged the record-carrying commit into c2d-memory-store.
  moetemp
  committed
  d3c4a5f
- Merged the record-carrying commit into c3a-redis-append.
  moetemp
  committed
  b1faed5
- Merged the record-carrying commit into c3b-redis-retention.
  moetemp
  committed
  53ffc1c
- Staged Workflow output before completing the Workflow Task.
  moetemp
  committed
  6b64a02
- Added owner checks and topic closes to the stream layer.
  moetemp
  committed
  d91847f
- Promoted output on an accepted completion and settled the rest from History.
  moetemp
  committed
  e0c769d
- Ended reads with their owner and refused reads past lost records.
  moetemp
  committed
  7ecf2be
- Closed a chain's streams after its final Workflow Task.
  moetemp
  committed
  4b85b8a
- Settled stages left behind from the reader, and tested both SDKs on one Redis.
  moetemp
  committed
  57e9aec
- Merged the stream layer through read expiry into c4a-stage-output.
  moetemp
  committed
  31cf04e
- Merged the stream layer through read expiry into c4b-promote-output.
  moetemp
  committed
  64a86e2
- Merged the stream layer through read expiry into c4c-close-chain.
  moetemp
  committed
  498ad2e
- Checked the Redis version and eviction policy on every primary.
  moetemp
  committed
  59413f0
- Deleted an owner's Redis streams across every primary.
  moetemp
  committed
  7ec395d
- Told an empty read from the beginning about records trimmed since.
  moetemp
  committed
  c51d4a3
- Merged the start watermark into c3e-reader-repair.
  moetemp
  committed
  2de4a31
- Merged the start watermark into c3f-server-checks.
  moetemp
  committed
  f683ea6
- Merged the start watermark into c3g-owner-delete.
  moetemp
  committed
  37849a3
- Served the store service in process for lang.
  moetemp
  committed
  43e3b9a
- Added stream store calls to the C bridge.
  moetemp
  committed
  40ea97c
- Carried lang's batch digest on the stream append request.
  moetemp
  committed
  f549c38
- Merged the append digest field into c2b-streams-crate.
  moetemp
  committed
  af7bd39
- Left the append digest to lang and refused one that is missing.
  moetemp
  committed
  c4654a4
- Merged the lang digest into c3c-owner-checks.
  moetemp
  committed
  a0276bf
- Merged the lang digest into c2c-reader-supersession.
  moetemp
  committed
  20b7666
- Merged the lang digest into c2d-memory-store.
  moetemp
  committed
  113bbb4
- Merged the lang digest into c3a-redis-append.
  moetemp
  committed
  b3a2ee7
- Merged the lang digest into c3b-redis-retention.
  moetemp
  committed
  f8d8eed
- Passed lang's append digest through to the store.
  moetemp
  committed
  46db6f8
- Carried a Workflow's topic closes in its output commit.
  moetemp
  committed
  a9c95c9
- Merged the lang digest into c3d-read-expiry-and-end.
  moetemp
  committed
  0b33a14
- Merged the lang digest into c3e-reader-repair.
  moetemp
  committed
  a0d07bd
- Ran the cross-implementation dedupe through the stream layer.
  moetemp
  committed
  a559c93
- Merged the lang digest into c3f-server-checks.
  moetemp
  committed
  460c193
- Merged the lang digest into c3g-owner-delete.
  moetemp
  committed
  3986ea8
- Merged the commit closes into c1b-output-marker.
  moetemp
  committed
  cf26fc7
- Refused a stream close the marker could not prove.
  moetemp
  committed
  632a591
- Merged the commit closes into c1c-output-replay.
  moetemp
  committed
  c16ab29
- Merged the commit closes into c2a-stream-protos.
  moetemp
  committed
  fc0535e
- Merged the commit closes into c2b-streams-crate.
  moetemp
  committed
  d6893bd
- Merged the commit closes into c2c-reader-supersession.
  moetemp
  committed
  4365aa7
- Merged the commit closes into c2d-memory-store.
  moetemp
  committed
  01be95f
- Merged the commit closes into c3a-redis-append.
  moetemp
  committed
  16ace44
- Merged the commit closes into c3b-redis-retention.
  moetemp
  committed
  ca7745a
- Merged the commit closes into c3c-owner-checks.
  moetemp
  committed
  6462f66
- Merged the commit closes into c3d-read-expiry-and-end.
  moetemp
  committed
  9eaa19b
- Merged the commit closes into c3e-reader-repair.
  moetemp
  committed
  42349c8
- Merged the commit closes into c3f-server-checks.
  moetemp
  committed
  3f31ef5
- Merged the commit closes into c3g-owner-delete.
  moetemp
  committed
  0a2227f
- Tested that a failed Workflow Task stages nothing.
  moetemp
  committed
  d67f669
- Merged the failed-task test into c4b-promote-output.
  moetemp
  committed
  780a187
- Merged the failed-task test into c4c-close-chain.
  moetemp
  committed
  e553375
- Tested a retried and a cron successor keeping the chain open.
  moetemp
  committed
  6729028
- Merged the stream layer through owner delete into c4a-stage-output.
  moetemp
  committed
  a71ed9c
- Merged the stream layer through owner delete into c4b-promote-output.
  moetemp
  committed
  8a26861
- Merged the stream layer through owner delete into c4c-close-chain.
  moetemp
  committed
  cd17e55
- Merged the stream layer through owner delete into c5a-stream-service.
  moetemp
  committed
  95268cb
- Sent owner deletes through the stream layer.
  moetemp
  committed
  8b4fe0c
- Tested that an owner delete is checked before the store.
  moetemp
  committed
  92c8d85
- Merged the stream layer through owner delete into c5b-c-bridge-streams.
  moetemp
  committed
  2db49ec
- Closed a Workflow's topics in the store after promoting its output.
  moetemp
  committed
  015c1d3
- Merged the topic closes after promotion into c4c-close-chain.
  moetemp
  committed
  e180b04
- Merged the topic closes after promotion into c5a-stream-service.
  moetemp
  committed
  c4397ea
- Merged the topic closes after promotion into c5b-c-bridge-streams.
  moetemp
  committed
  0bc9690
- Closed a stage's topics in commit order and gave up on a refusal.
  moetemp
  committed
  57f5116
- Merged the ordered topic closes into c4c-close-chain.
  moetemp
  committed
  84ba215
- Merged the ordered topic closes into c5a-stream-service.
  moetemp
  committed
  cbae2f6
- Merged the ordered topic closes into c5b-c-bridge-streams.
  moetemp
  committed
  844e4b2
- Closed the chain only after the final task's topic closes.
  moetemp
  committed
  47efc07
- Merged the chain close order into c5a-stream-service.
  moetemp
  committed
  69e04c6
- Merged the chain close order into c5b-c-bridge-streams.
  moetemp
  committed
  e327ea7
- Retried a failed stream promotion before its closes.
  moetemp
  committed
  8b3a1a5
- Capped the producers a read remembers.
  moetemp
  committed
  373680f
- Merged the producer cap into c2d-memory-store.
  moetemp
  committed
  7a6a581
- Refused a closed chain's append before creating its topic.
  moetemp
  committed
  45011e9
- Merged the review fixes into c3a-redis-append.
  moetemp
  committed
  504aadf
- Merge branch 'moe/brook/c4b-promote-output' into moe/brook/c4c-close-chain
  moetemp
  committed
  23865e6
- Tested the chain close after a failed final promotion.
  moetemp
  committed
  4715b3c
- Merge branch 'moe/brook/c4c-close-chain' into moe/brook/c5a-stream-service
  moetemp
  committed
  04af1c2
- Merge branch 'moe/brook/c5a-stream-service' into moe/brook/c5b-c-bridge-streams
  moetemp
  committed
  068873e
- Made Redis failures that clear up retryable and bounded blocked reads.
  moetemp
  committed
  5dbac6a
- Merged the review fixes into c3b-redis-retention.
  moetemp
  committed
  0182544
- Widened the retention test margins.
  moetemp
  committed
  e14981a
- Merged the review fixes into c3c-owner-checks.
  moetemp
  committed
  dd0da64
- Checked the topic close in the append script itself.
  moetemp
  committed
  7785a37
- Merged the review fixes into c3d-read-expiry-and-end.
  moetemp
  committed
  1ed9dca
- Let an empty read survive a quiet log coming back.
  moetemp
  committed
  36e8efa
- Kept a stage undecided when a completion is answered NotFound.
  moetemp
  committed
  66f7715
- Ended a read only once its chain has no stage pending.
  moetemp
  committed
  d734439
- Merged the review fixes into c3e-reader-repair.
  moetemp
  committed
  65d1ea8
- Gave the conformance owner fake the History calls repair makes.
  moetemp
  committed
  717a214
- Repaired the chain when a read ends on its close.
  moetemp
  committed
  880f742
- Went past stages the store refuses for good in repair.
  moetemp
  committed
  8fb6b36
- Remembered undecided stages for a few seconds.
  moetemp
  committed
  f92808a
- Merged the review fixes into c3f-server-checks.
  moetemp
  committed
  27ac241
- Kept a run's promotions in commit order behind a failed one.
  moetemp
  committed
  4fd590e
- Allowed CLUSTER SLOTS in the documented ACL.
  moetemp
  committed
  cf537ab
- Merged the review fixes into c3g-owner-delete.
  moetemp
  committed
  2ede8aa
- Said a delete during a reshard should run again.
  moetemp
  committed
  63a26af
- Merge branch 'moe/brook/c4b-promote-output' into moe/brook/c4c-close-chain
  moetemp
  committed
  f3f4f6b
- Closed the chain on every chain end, waiting on all of the run's work.
  moetemp
  committed
  09704ca
- Merge branch 'moe/brook/c4c-close-chain' into moe/brook/c5a-stream-service
  moetemp
  committed
  5926873
- Merge branch 'moe/brook/c5a-stream-service' into moe/brook/c5b-c-bridge-streams
  moetemp
  committed
  67360df
- Capped a stream read at 1,000 records and a 60 s wait.
  moetemp
  committed
  629e3ae
- Merge branch 'moe/brook/c5a-stream-service' into moe/brook/c5b-c-bridge-streams
  moetemp
  committed
  88cc7ad
- Gave a cancelled stream read its own failure message.
  moetemp
  committed
  99e600f
- Fixed the stream record body comment and set its Ruby package.
  moetemp
  committed
  d46e6b0
- Merge remote-tracking branch 'moedash/moe/brook/c1a-output-protos' into work/c1b
  moetemp
  committed
  7a2292a
- Refused output records with a bad topic or over the byte budget.
  moetemp
  committed
  bfbd726
- Noted that the marker boundary covers one activation.
  moetemp
  committed
  6d928c1
- Merge remote-tracking branch 'moedash/moe/brook/c1b-output-marker' into work/c1c
  moetemp
  committed
  3d54354
- Left the store format out of the replay compare.
  moetemp
  committed
  2b6143b
- Merge remote-tracking branch 'moedash/moe/brook/c1c-output-replay' into moe/brook/c2a-stream-protos
  moetemp
  committed
  3c75984
- Corrected the digest comment and stated Core's read caps.
  moetemp
  committed
  59747df
- Merge branch 'moe/brook/c2a-stream-protos' into moe/brook/c2b-streams-crate
  moetemp
  committed
  e54854a
- Merged the Core review fixes below into c2c-reader-supersession.
  moetemp
  committed
  27935ab
- Merged the Core review fixes below into c2d-memory-store.
  moetemp
  committed
  3e3e919
- Merged the Core review fixes below into c3a-redis-append.
  moetemp
  committed
  0f846a4
- Merged the Core review fixes below into c3b-redis-retention.
  moetemp
  committed
  83ddcba
- Merged the Core review fixes below into c3c-owner-checks.
  moetemp
  committed
  e275099
- Merged the Core review fixes below into c3d-read-expiry-and-end.
  moetemp
  committed
  9e609cb
- Merged the Core review fixes below into c3e-reader-repair.
  moetemp
  committed
  8ff1274
- Merged the Core review fixes below into c3f-server-checks.
  moetemp
  committed
  e48aaa0
- Merged the Core review fixes below into c3g-owner-delete.
  moetemp
  committed
  8ed0ded
- Used the shared topic rules in the stream layer.
  moetemp
  committed
  f809ae0
- Merged the shared topic rules into c3d-read-expiry-and-end.
  moetemp
  committed
  a3df602
- Merge branch 'moe/brook/c2b-streams-crate' into moe/brook/c4a-stage-output
  moetemp
  committed
  1f921dc
- Merge branch 'moe/brook/c4c-close-chain' into moe/brook/c5a-stream-service
  moetemp
  committed
  39ab70b
- Merge branch 'moe/brook/c5a-stream-service' into moe/brook/c5b-c-bridge-streams
  moetemp
  committed
  292a2dc
- Merge branch 'moe/brook/c4a-stage-output' into moe/brook/c4b-promote-output
  moetemp
  committed
  1b6d5fd
- Merge branch 'moe/brook/c4b-promote-output' into moe/brook/c4c-close-chain
  moetemp
  committed
  8ba3b73
- Noted that each memory store connect stands alone.
  moetemp
  committed
  b20277e
- Merge branch 'moe/brook/c5a-stream-service' into moe/brook/c5b-c-bridge-streams
  moetemp
  committed
  68892f2
- Merged the shared topic rules into c3e-reader-repair.
  moetemp
  committed
  c976bb2
- Merged the shared topic rules into c3f-server-checks.
  moetemp
  committed
  740f354
- Merged the shared topic rules into c3g-owner-delete.
  moetemp
  committed
  0d55a46
- Merged the Core review fixes through owner delete into c4a-stage-output.
  moetemp
  committed
  3c21c11
- Merge branch 'moe/brook/c4a-stage-output' into moe/brook/c4b-promote-output
  moetemp
  committed
  01288d1
- Merge branch 'moe/brook/c4b-promote-output' into moe/brook/c4c-close-chain
  moetemp
  committed
  8b35a1a
- Merge branch 'moe/brook/c4c-close-chain' into moe/brook/c5a-stream-service
  moetemp
  committed
  d27daba
- Merge branch 'moe/brook/c5a-stream-service' into moe/brook/c5b-c-bridge-streams
  moetemp
  committed
  d36bb30
- Explained why a refused promotion is retried.
  moetemp
  committed
  de291d8
- Merge branch 'moe/brook/c4b-promote-output' into moe/brook/c4c-close-chain
  moetemp
  committed
  404865f
- Merge branch 'moe/brook/c4c-close-chain' into moe/brook/c5a-stream-service
  moetemp
  committed
  edae88d
- Merge branch 'moe/brook/c5a-stream-service' into moe/brook/c5b-c-bridge-streams
  moetemp
  committed
  5ae0ae7
- Served the stream service's Nexus calls in Core.
  moetemp
  committed
  24c3b13
- Moved the Nexus calls to their own service and added INVALID_ARGUMENT.
  moetemp
  committed
  c1d835f
- Ported the Nexus service store cases from Python to Core.
  moetemp
  committed
  81b203b
- Added the long-poll state field and the token model to the contract.
  moetemp
  committed
  9c46c54
- Merged the N0b store-case ports into cn4-nexus-stream-calls.
  moetemp
  committed
  0b988b2
- Ported the Nexus service store cases from Python to Core.
  moetemp
  committed
  cb5cc04
- Covered the table's catch-all rows in the failure test.
  moetemp
  committed
  379e809
- Merged the remote cn4-nexus-stream-calls.
  moetemp
  committed
  7ace8d7
- Made a record failure at the Nexus edge non-retryable.
  moetemp
  committed
  c64e3f2
- Moved the streams crate to the workspace's base64 version.
  moetemp
  committed
  bba0fbb
- Bounded what a Nexus read takes from its caller and its codec.
  moetemp
  committed
  1981843
- Said the read state's 64 KiB budget is raw bytes.
  moetemp
  committed
  8835dc5
Commits on Oct 13, 2026
- Added the output stream commit command with records and closes.
  moetemp
  committed
  591fb6f
- Recorded stream output in a marker before other commands.
  moetemp
  committed
  6515d9e
- Validated stream output on replay from a Core-built manifest.
  moetemp
  committed
  57fce3c
- Staged and promoted Workflow output in the stream stores.
  moetemp
  committed
  7992f35
- Settled left-behind stages from the reader.
  moetemp
  committed
  4d49380
- Staged Workflow output before completing the Workflow Task.
  moetemp
  committed
  f1259db
- Promoted or aborted Workflow output by the completion's answer.
  moetemp
  committed
  418913c
- Closed a chain's streams after its final Workflow Task.
  moetemp
  committed
  1da9cd6
- Gave each Worker the process's stream store.
  moetemp
  committed
  c1b68b3
- Merged the Ford Core top into cn4.
  moetemp
  committed
  3b45664
- Merged the re-cut Ford Core top into cn4.
  moetemp
  committed
  1859d5b
- Merged the write-through Ford Core top into cn4.
  moetemp
  committed
  c6dda84

Commits

Commits on Oct 9, 2026

Commits on Oct 10, 2026

Commits on Oct 11, 2026