Skip to content

feat: continue an operation only as its initiator or an operator - #582

Merged
wchwawa merged 4 commits into
mainfrom
feat/initiator-enforcement
Oct 24, 2026
Merged

wchwawa merged 4 commits into
mainfrom
feat/initiator-enforcement

Conversation

@wchwawa

@wchwawa wchwawa commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

TLDR

A durable operation is now continued, aborted, or completed only by the
principal that began it or by a principal holding the operator role on the
root; any other principal's request is refused and changes nothing. Operation
status and change events name the initiator over the wire and in the CLI.
This is the second half of initiator attribution (R4 of #566), on the records
#581 writes. Stacked on #581.

Problem

Relates to #566.

#581 records who began each operation, but nothing read the record. Any
writer on a root could finalize, abort, heartbeat, or clean up another
principal's in-flight publication, continue its commit or restore, renew or
retire its snapshot, or abort its index registration, all under its own
credential, and the durable records then showed one principal's operation
advanced by another with no refusal anywhere. Observers had no way to ask who
began an operation: the status and the change events did not say.

Consequences

On a shared root the authorization of #578 stopped at the request kind: a
writer could do anything a writer may do, including to operations other
writers had in flight. An Agent's multi-request publication or commit could
be aborted or finished by any peer with the role, and an audit of a change
event could not name the actor without the owner's logs.

Example

Principal db8b…f4eb renews a snapshot principal c8e3…e78d minted, as the
TLS gate records it:

nokv ... workbench workbench_snapshot_renew '{"id":"tls","name":"t11","ttl_days":2}'
{"class":"operator_required","code":"PermissionDenied","next_action":"operator",
 "message":"principal db8b82006dad4b5af39b73c4c9d7f4eb did not begin this
 operation; principal c8e34f24e9ad86e902d91771a8dbe78d did, and only it or an
 operator continues it","retryable":false,"status":"error"}

The snapshot is unchanged; its minter, or an operator, renews it. Any reader
of the root sees who began an operation:

nokv ... operation status e46322f02ec2646423c7d3713c895ebc
{"operation":"publish","initiator":"c8e34f24e9ad86e902d91771a8dbe78d",
 "observed_state":"Succeeded","receipt":{…},…}

Solution

  • One rule. WriteAuthority::continuation decides whether a request may
    continue an operation that initiator began: its own, one nobody is
    recorded for, or any one as an operator; the owner's own work (recovery,
    cleanup, GC) continues everything. The rule runs at the one place each
    service loads or verifies the current operation: the publication record
    check behind every publish continuation, the commit build and restore
    loaders behind every step, the index registration loader, and snapshot
    renewal and retirement. A later append attempt is a continuation of the
    parent it extends. Snapshot consumers and the reaping of an expired
    snapshot are not continuations and stay open to the role.
  • Content-derived ids. Commit and restore operation ids derive from
    content, so another principal beginning the same commit or restore
    observes the first one: while it is in flight the request is refused with
    a retryable Conflict (OperationHeldByAnother), and once it is terminal
    the request replays its outcome. Publish, append, snapshot, and index ids
    are caller-chosen and already sealed with the initiator by feat: record the initiating principal in workspace format 15 #581.
  • Server. The executor receives the role the server authorized along
    with the principal, so the operator override applies; a refused
    continuation maps to PermissionDenied, not retryable.
  • Exposure. OperationStatus.initiator and ChangeEvent.initiator join
    the wire contract as PrincipalIdentity (v15 is unreleased; no golden
    digest changed); nokv operation status and the Python SDK's
    operation_status report initiator as hex, or null on a plaintext
    deployment, so the two surfaces keep reporting one status.
  • Gates. The TLS gate gains an operator principal, a fourth writer, and
    five scenarios: an operation id presented by another principal with the
    same inputs (answered from the existing operation, nothing written) and
    with other inputs (identity_conflict), a status read by another
    principal, a snapshot renewed and retired across principals, an identical
    commit by two principals (replayed), and four principals writing one root
    at once.

Gain

A principal's in-flight work is its own: no peer with the same role can
finish, abort, or extend it, and the refusal is typed and leaves nothing
behind. An operator can still unblock anything. Everyone who can read a root
can see who began each operation and each change.

Tradeoffs

  • Operator override is unconditional. An operator continues any
    operation in any phase; there is no narrower "abort only" grant.
  • Reads are not restricted. Any reader of the root sees every
    operation's status and initiator; the role model already lets a reader see
    the root's content.
  • A held commit is a client-side wait. The retryable Conflict tells
    the second principal to retry; nothing on the owner wakes it when the first
    commit completes.
  • Commit retirement records no initiator. It is begun by the owner's
    lifecycle, not by a request, so nothing checks it.
  • Same inputs are observed, not refused. A principal presenting another
    principal's operation id with identical inputs is answered from the
    existing operation by the client's replay-by-observation, exactly as the
    first principal would be; nothing is written and the status still names
    the first principal. The owner refuses only a write under that id.
  • Plaintext deployments. Nothing is enforced where nobody is
    authenticated, as before.

Validation

All runs used Rust 1.99.1 on macOS:

  • cargo fmt --all -- --check: clean.
  • cargo clippy --workspace --all-targets -- -D warnings: clean.
  • cargo test --locked --no-fail-fast --workspace: 1,345 passed, 0 failed,
    16 ignored. New tests: the rule itself; per service, a continuation by
    another principal refused, by the initiator and by an operator allowed
    (publication, commit, restore, index registration, snapshot); an identical
    commit and restore held while in flight and the commit replayed once
    terminal; over RPC, a publication abort and a snapshot renewal refused
    with PermissionDenied and allowed for an operator; status and change
    events naming the initiator; the CLI status JSON naming it.
  • etcd identity tests against a local etcd 3.7.1: 3 passed.
  • Live gates on the debug binary with moto: rpc_tls_gate.py 15 of 15
    (new: t9 to t13), cli_contract_gate.py 6 of 6.
  • Python: the direct SDK contract gate (python_contract_gate.py, on a
    debug wheel built with maturin, against moto) 6 of 6; CI's first run
    against main found the SDK and CLI statuses no longer matching before the
    SDK reported initiator. The two gate unit-test suites pass (7 tests).
  • A/B against the feat: record the initiating principal in workspace format 15 #581 debug binary over TLS, 30 rotated rounds per arm
    plus 600 pooled-session requests, paired medians: stat +0.02 ms,
    4 KiB put -0.37 ms, pooled session stat 0.26 ms on both arms. The
    continuation check is one comparison per step; no cost is measurable.

Not run:

  • Fault injection. A commit abandoned mid-flight by one principal and
    retried by another is covered at the metadata layer, not with a crashed
    owner.

Size: 1,044 changed lines, under the 5,000-line review threshold.

Every continuation, abort, and completion of a durable operation now
checks the request's authority against the initiator the operation
records: the initiator continues its own work, a principal holding the
operator role on the root continues any work, and the owner's own
recovery and cleanup continue everything. A request by any other
principal is refused as NotTheInitiator and changes nothing. The check
sits at the one place each service loads or verifies the current
operation: publication, commit, restore, Generic index registration,
and snapshot renewal and retirement. A later append attempt is a
continuation of the parent it extends.

Commit and restore operation ids derive from content, so another
principal beginning the same commit or restore observes the first one:
its terminal outcome replays, and while it is in flight the request is
refused as OperationHeldByAnother, for the caller to retry.

Relates to #566.

Signed-off-by: wchwawa <wch19961116@gmail.com>
…initiator

The server passes the role it authorized along with the principal, so
the executor knows when the operator override applies, and maps a
refused continuation to PermissionDenied and a held content-derived
operation to a retryable Conflict. Operation status and change events
carry the initiator over the wire as a PrincipalIdentity, the CLI's
operation status prints it as initiator, and the TLS gate exercises the
rule end to end: a reused operation id, a status read by another
principal, a snapshot renewed and retired across principals, an
identical commit by two principals, and four principals writing one
root at once.

Relates to #566.

Signed-off-by: wchwawa <wch19961116@gmail.com>
@wchwawa
wchwawa requested a review from feichai0017 as a code owner October 23, 2026 16:11
@feichai0017 feichai0017 moved this to In Progress in NoKV Delivery Oct 23, 2026
@wchwawa
wchwawa changed the base branch from feat/initiator-attribution to main October 24, 2026 05:45
Signed-off-by: wchwawa <wch19961116@gmail.com>
The SDK's operation_status reports the same fields as the CLI's operation
status, which the direct Python SDK contract gate holds it to; it now
names the initiator the owner recorded, or None on a plaintext deployment.

Relates to #566.

Signed-off-by: wchwawa <wch19961116@gmail.com>
@wchwawa
wchwawa merged commit fcf3d6a into main Oct 24, 2026
11 checks passed
@wchwawa
wchwawa deleted the feat/initiator-enforcement branch October 24, 2026 06:10
@github-project-automation github-project-automation Bot moved this from In Progress to Done in NoKV Delivery Oct 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants