Repository navigation
feat: continue an operation only as its initiator or an operator - #582
Merged
Merged
Conversation
Every continuation, abort, and completion of a durable operation now checks the request's authority against the initiator the operation records: the initiator continues its own work, a principal holding the operator role on the root continues any work, and the owner's own recovery and cleanup continue everything. A request by any other principal is refused as NotTheInitiator and changes nothing. The check sits at the one place each service loads or verifies the current operation: publication, commit, restore, Generic index registration, and snapshot renewal and retirement. A later append attempt is a continuation of the parent it extends. Commit and restore operation ids derive from content, so another principal beginning the same commit or restore observes the first one: its terminal outcome replays, and while it is in flight the request is refused as OperationHeldByAnother, for the caller to retry. Relates to #566. Signed-off-by: wchwawa <wch19961116@gmail.com>
…initiator The server passes the role it authorized along with the principal, so the executor knows when the operator override applies, and maps a refused continuation to PermissionDenied and a held content-derived operation to a retryable Conflict. Operation status and change events carry the initiator over the wire as a PrincipalIdentity, the CLI's operation status prints it as initiator, and the TLS gate exercises the rule end to end: a reused operation id, a status read by another principal, a snapshot renewed and retired across principals, an identical commit by two principals, and four principals writing one root at once. Relates to #566. Signed-off-by: wchwawa <wch19961116@gmail.com>
Signed-off-by: wchwawa <wch19961116@gmail.com>
The SDK's operation_status reports the same fields as the CLI's operation status, which the direct Python SDK contract gate holds it to; it now names the initiator the owner recorded, or None on a plaintext deployment. Relates to #566. Signed-off-by: wchwawa <wch19961116@gmail.com>
This was referenced Oct 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
A durable operation is now continued, aborted, or completed only by the
principal that began it or by a principal holding the operator role on the
root; any other principal's request is refused and changes nothing. Operation
status and change events name the initiator over the wire and in the CLI.
This is the second half of initiator attribution (R4 of #566), on the records
#581 writes. Stacked on #581.
Problem
Relates to #566.
#581 records who began each operation, but nothing read the record. Any
writer on a root could finalize, abort, heartbeat, or clean up another
principal's in-flight publication, continue its commit or restore, renew or
retire its snapshot, or abort its index registration, all under its own
credential, and the durable records then showed one principal's operation
advanced by another with no refusal anywhere. Observers had no way to ask who
began an operation: the status and the change events did not say.
Consequences
On a shared root the authorization of #578 stopped at the request kind: a
writer could do anything a writer may do, including to operations other
writers had in flight. An Agent's multi-request publication or commit could
be aborted or finished by any peer with the role, and an audit of a change
event could not name the actor without the owner's logs.
Example
Principal
db8b…f4ebrenews a snapshot principalc8e3…e78dminted, as theTLS gate records it:
The snapshot is unchanged; its minter, or an operator, renews it. Any reader
of the root sees who began an operation:
Solution
WriteAuthority::continuationdecides whether a request maycontinue an operation that
initiatorbegan: its own, one nobody isrecorded for, or any one as an operator; the owner's own work (recovery,
cleanup, GC) continues everything. The rule runs at the one place each
service loads or verifies the current operation: the publication record
check behind every publish continuation, the commit build and restore
loaders behind every step, the index registration loader, and snapshot
renewal and retirement. A later append attempt is a continuation of the
parent it extends. Snapshot consumers and the reaping of an expired
snapshot are not continuations and stay open to the role.
content, so another principal beginning the same commit or restore
observes the first one: while it is in flight the request is refused with
a retryable
Conflict(OperationHeldByAnother), and once it is terminalthe request replays its outcome. Publish, append, snapshot, and index ids
are caller-chosen and already sealed with the initiator by feat: record the initiating principal in workspace format 15 #581.
with the principal, so the operator override applies; a refused
continuation maps to
PermissionDenied, not retryable.OperationStatus.initiatorandChangeEvent.initiatorjointhe wire contract as
PrincipalIdentity(v15 is unreleased; no goldendigest changed);
nokv operation statusand the Python SDK'soperation_statusreportinitiatoras hex, ornullon a plaintextdeployment, so the two surfaces keep reporting one status.
five scenarios: an operation id presented by another principal with the
same inputs (answered from the existing operation, nothing written) and
with other inputs (
identity_conflict), a status read by anotherprincipal, a snapshot renewed and retired across principals, an identical
commit by two principals (replayed), and four principals writing one root
at once.
Gain
A principal's in-flight work is its own: no peer with the same role can
finish, abort, or extend it, and the refusal is typed and leaves nothing
behind. An operator can still unblock anything. Everyone who can read a root
can see who began each operation and each change.
Tradeoffs
operation in any phase; there is no narrower "abort only" grant.
operation's status and initiator; the role model already lets a reader see
the root's content.
Conflicttellsthe second principal to retry; nothing on the owner wakes it when the first
commit completes.
lifecycle, not by a request, so nothing checks it.
principal's operation id with identical inputs is answered from the
existing operation by the client's replay-by-observation, exactly as the
first principal would be; nothing is written and the status still names
the first principal. The owner refuses only a write under that id.
authenticated, as before.
Validation
All runs used Rust 1.99.1 on macOS:
cargo fmt --all -- --check: clean.cargo clippy --workspace --all-targets -- -D warnings: clean.cargo test --locked --no-fail-fast --workspace: 1,345 passed, 0 failed,16 ignored. New tests: the rule itself; per service, a continuation by
another principal refused, by the initiator and by an operator allowed
(publication, commit, restore, index registration, snapshot); an identical
commit and restore held while in flight and the commit replayed once
terminal; over RPC, a publication abort and a snapshot renewal refused
with
PermissionDeniedand allowed for an operator; status and changeevents naming the initiator; the CLI status JSON naming it.
rpc_tls_gate.py15 of 15(new: t9 to t13),
cli_contract_gate.py6 of 6.python_contract_gate.py, on adebug wheel built with maturin, against moto) 6 of 6; CI's first run
against main found the SDK and CLI statuses no longer matching before the
SDK reported
initiator. The two gate unit-test suites pass (7 tests).plus 600 pooled-session requests, paired medians:
stat+0.02 ms,4 KiB
put-0.37 ms, pooled sessionstat0.26 ms on both arms. Thecontinuation check is one comparison per step; no cost is measurable.
Not run:
retried by another is covered at the metadata layer, not with a crashed
owner.
Size: 1,044 changed lines, under the 5,000-line review threshold.