refactor(provider-services): retire process-family typed ops behind the broker envelope seam (U10-U14) - #540
Merged
Merged
Conversation
New check-async-gate subcommand rejects blocking calls in async worker contexts across broker, d2bd, and provider crates: lexical probe mirroring provider_crate_policy probes, reusing clippy.toml disallowed-methods as the single blocking-fn source, spawn_blocking bodies exempt per KTD9, named canonical-JSON violations. Fixture-verified only; wiring into the enforcement chain lands with U13's sweep on the migrated tree.
EffectServiceSupervisor + EffectServiceActor mirroring ResourceManager supervision: spawn_linked children under a per-zone supervisor, respawn from durable rows, ractor requeue timers, generational revision counter bumped on respawn/republish with StaleRevision refusal (KTD5). Fixture trait only; rendezvous binding lands in U8 integration.
SCM_RIGHTS carriage on request/response frames: fd-index fields on ForwardOperationRequest/Outcome validated against declared indexes, MAX_FRAME_FDS=8 ceiling with pre-dispatch row-facet validation (max_fds + fd_kind on committed rows, all rows regenerated), FD_LEG closed-set peer refusal code shared across both legs, envelope + forwarding + rendezvous plumbing, toolkit invoke_named_with_fds, LoopbackPeer + cross-socket fd round-trips and mismatch/oversize/kind/mint refusals (AE1).
…14 mechanism) Four new gates: family/type/role/provider-ref literals incl. broker dispatch arms and runtime-assembled names, ServerState references in d2bd, generated-view producer provenance, and d2b-broker provider-free manifest pin. SHARED_FAMILY_KNOWLEDGE_RATCHET seeded from the current tree with per-row retires_with documentation (U10-U12 delete rows). Structural bazel-visibility edge designed but deferred: no bazel binary on this host; source probes are the working enforcement until U14 final integration wires the visibility edge.
Generic CellStore keyed (cell, invocation_id, initiating_principal) replaces lifecycle_leases and runner_pidfd_registry statics: durable one-time cells commit before the effect with unknown-outcome reconcile (AE2 restart-replay resistance), ephemeral cells keep reset-on-restart, retention caps exempt consumed one-time markers. Durability facet on committed rows (stateCell + cell_durability), generator regenerated 70 rows byte-stably (KTD3).
ForwardContext block (epoch, zone, provider-set revision, controller and guest generations, initiating identity, deadline budget) minted only by the broker from a durable monotonic cache; daemon publishes values over the origination leg; rendezvous enforces field-wise with the STALE_CONTEXT code, serves the context deadline as the per-call handler deadline, and rejects every pre-restart context via strictly increasing broker-epoch nonces (KTD2, KTD4 transport).
Closed peer-code set grown to 13 (handler-refused/errored/timed-out/ crashed, stale-wire-version; FD_LEG, STALE_CONTEXT from U1/U2): dispatch failures keep their own code, nothing flattens to unregistered-handler outside genuine missing-handler cases. Local handlers run as abortable tokio tasks on a dedicated 2-worker set; panic -> HANDLER_CRASHED closing audit refusal; deadline expiry -> HANDLER_TIMED_OUT abort while innocent calls answer; blocking pool untouched (KTD7).
Daemon publishes provider-set revision + zone generations over the origination leg: committed PublishTrustedContext row + wire variant, runtime dispatch arm with audit, TrustedContextStore publish/ack; daemon side fires from ForwardRendezvous::publish after the revision assignment (composition.rs zero edits), publish_generations and set_broker_epoch now production-called with the ACKED epoch, transport errors leave the rendezvous fail-closed. Guest generation publishes the documented fleet base. Bazel d2bd build green; views regenerated byte-stable.
All 60 d2b-provider-* packages drop //visibility:public for closed enumerated __pkg__ sets anchored on d2bd (daemon composition root) and legit provider-to-provider consumers; broker and shared crates are absent from every set (enforcement proven by a failing visibility build), exports_files preserved for data-graph consumers. Broker composition crate (U6) joins the sets when it lands.
ProviderSet::start spawns the per-zone EffectServiceSupervisor after the plane publishes declared services: one linked actor per declared service, respawn from durable rows, requeue timers, generational revisions (republish bumps, stale bindings refuse StaleRevision). Composition seam with_effect_service_factory / resolve / publish_effect_service; named refusals for duplicate/factory-missing services. Fixed harness restart recovery bug (recovered rows were never recorded, killing a recovered actor never respawned). Rendezvous binding consumption is U8b.
Closed DeadlineTier facet (Standard 25s / Extended 60s) on committed rows, minted into the context and binding both legs under the ceiling; forwarded dispatch runs on its own task with panic -> handler-crashed by name, never a dropped socket; retired-variant gating runs Value-first before typed decode (stale-wire-version refusal + audit record), with closed RETIRED_WIRE_VARIANTS table (empty in production; U10 fills it) and a mixed-version matrix fixture (KTD4/KTD7/KTD10).
…(U5) Daemon-native host-ownership preflight migrates off its typed placeholder onto BrokerEnvelope::call with CallerAuthority::Daemon over the origination-leg carrier, forward-budget timeout mapping to internal-broker-timeout (never unregistered-handler); prebind gap documented fail-closed. Trusted-context store init made lazy: opens on first PublishTrustedContext, open failure refuses that publication while the broker keeps serving (fixes socket-activation startup regression); restart-epoch invalidation preserved. OwnershipMatrixCheck row promoted live; placeholder pins in production_binary_policy superseded.
d2b-broker is lib-only and provider-free; new d2b-broker-composition links provider handler crates, preserves the d2b-broker binary name (socket service units + bazel refs repointed, Nix host-tools rewired, guest workspace + lock regenerated). Routing rule admits zero in-broker ops this pass (fail-closed seam::verify_startup_routing, empty production registration); dependency-surface audit scans handler crates for syscall/asm!/ctor/registration surfaces with fixture crates (31/31 tests). Broker startup no longer depends on provider code.
Forwarded operations resolve to the declaring effect service via the committed operation facet (ResolveOperation, deterministic winner) and dispatch to the hosted actor over the normal carrier; binding revision captured at call start, mid-call death or republish refuses with the dedicated stale-revision code, never a hang (KTD5). Declined maps to handler-refused; OperationUnserved falls through to provider tables. Blanket dead-code allowance drained; per-seam allowances marked for U9+.
…(U7) ServiceDecl.methods becomes per-method contract facets: operation, payload schema, fd contract, state cells, privileges, deadline tier; zone-plane facets untouched (R4). Committed rows carry service+method resolution; generator closed-set validation refuses unresolvable/ambiguous/ malformed links and surfaces Service/Method in the triage view. Both generators regenerated byte-stable; drift gates green (KD6).
EvidenceChain roots and correlation records keyed (invocation_id, depth) with MAX_NESTED_DEPTH=8 and the nested-depth-exceeded closed refusal; nested calls authorize under the chain's initiating principal (graft rule), never the daemon class; serves_locally gating writes exactly one root record per invocation per executing leg — forwarded ops audit daemon-side only, in-broker broker-side only, mixed-leg chains never double-write (KTD6).
…ditive) 11 resource-agnostic kernel rows (open-pidfd, spawn-process, cgroup and reap surfaces) served in-broker by the KernelDispatcher at serve time; FdKind::Any for anon-inode pidfds; EnvelopeInvoke generic daemon->broker carrier; nix privileges-json drift gate (nix_surface_test pattern) pins brokerOperations vs the catalog; generator comma regression fixed+ fixtured. Provider-side migration and typed-arm retirement follow in the U10 cut.
…he envelope seam (U10) The ten process-family operations (OpenPidfd, OpenPeerPidfdFromAcceptedSocket, ObserveRunner, PollChildReaped, PrepareRuntimeDir, PrepareStateDir, CgroupKill, SignalRunner, DeregisterRunnerPidfd, SpawnRunner) retire their typed BrokerRequest/BrokerResponse wire variants at wire v6 and become the first forwarded-family pilot on the generic EnvelopeInvoke carrier: - Wire: the ten variants leave the enum; WIRE_VARIANTS drops them and RETIRED_WIRE_VARIANTS names them at v6, so a straggler peer gets the typed stale-wire-version refusal plus an audit record, never a silent drop (KTD10). PROTOCOL_VERSION stays 6: every surviving op's frame is byte-identical. - Catalog: the ten rows stay committed (declaringProvider d2b-provider-process, wireVariant dropped); eleven broker-generic kernel rows land in-broker and are served by kernel_ops.rs through KernelDispatcher (spawn-process, open-pidfd, open-peer-pidfd-from-accepted-socket, poll-child-reaped, prepare-directory, kill-cgroup, signal-pidfd, deregister-pidfd, delegate-cgroup-v2, open-cgroup-dir, observe-process). - Sandwich: the daemon-side family handlers (d2b-provider-process operations.rs) validate the typed payload and invoke the kernel as a NESTED EnvelopeInvoke over the origination socket (kernel_client.envelope_invoke_kernel), grafting on the U9 evidence chain (root id + ordered identities, KTD6). The daemon's legacy legs (open-pidfd, poll-child-reaped per runner, signal-pidfd with the tracked fd, deregister-pidfd, kill-cgroup via cgroupPath) use the kernels directly; supervisor BrokerProcessBackend migrated to envelope frames with the zone derived from the verified bundle. - Audit: the production envelope chain-audit sink is now wired at serve time (AuditLogChainSink -> AuditLog::write_chain_record), so every in-broker leg - incl. the nested kernel legs - is durable in the daily audit log keyed on the invocation id; inner legs are correlation records under the root id. - Ports: the retired arm's live behaviors were ported into the spawn-process kernel with tests (usbip device-bind extension, serving-worker ACL grant, stale cloud-hypervisor/video/otel socket cleanups, duplicate-runner refusal + registry wiring). Ratchet seeded for the kernel's family tokens (device-usbip, observability-otel, runtime-cloud-hypervisor) with census retirements. - Gates: matrix 7/7 (retired frames refused with typed code + audit), catalog completeness green, telemetry domain shrunk byte-stable, nix privileges rows kept (ops remain committed; drift gate green), ratchet green bidirectionally, generated-views drift 18/18, conformance 13/13 (cargo + bazel), and the new broker-seam-pilot cross-binary gate (tests/unit/gates/broker-seam-pilot.sh, bazel //tests/unit/gates:broker-seam-pilot) drives OpenPidfd end-to-end with fd liveness + audit continuity; it records a skip without root (production bundle posture), same lane as the host-integration gates. - Generator: operation rows now reject case-insensitive collisions (the toolkit envelope matches operation names case-insensitively). - Bazel: dep-surface synced for d2b-resource-types and d2b-provider-process. Deliberate gaps (documented in code): PollChildReaped returns empty notifications (the daemon's per-entry pidfd probe is the live drain); deregister-pidfd root calls report removed=false for legacy/adopted runners (daemon has no spawn invocation id); a future daemon launch path that passes escrow fds needs migration (spawn refuses inherited_fd_count>0); daemon-side durable chain mirror for forwarded legs is a tracked follow-up (DaemonAudit has no chain-record surface yet).
…l kernels (U11) The one-time guest lifecycle lease retires its last typed surface: the ConsumeLifecycleLease variant, arm, request/response structs, LifecycleLeaseOperation enum, audit shape, and the committed row all leave the tree (the row is DELETED, not variant-dropped - the op is gone from the catalog). The lease semantics now ride the generic cell mechanism as two broker-generic kernel rows served in-broker by the envelope dispatcher: - consume-cell / complete-cell: committed rows (owner broker-generic, host+guest profiles, authz d2bd/d2b-admin/d2b-launcher, destructive) that declare the lifecycle-leases cell with one-time durability - the row is the declaration surface (U3/KTD3), the kernel performs the compare-and- consume / completion against broker_store() with the principal-bound replay gate. The broker's lease-specific code (consume_lifecycle_lease, complete_lifecycle_lease, lifecycle_lease_identity, lifecycle_lease_cell, LIFECYCLE_LEASES_CELL, caller_principal) is deleted: zero lease code remains in the broker. - The daemon-side caller (composition.rs) now validates the lease fences itself (generation/policy-revision/operation-id shape, stop-only semantics) and invokes the two kernels over the origination socket via kernel_client.envelope_invoke_kernel, keying the one-time cell on the full lease identity (zone/guest/generations/policy/operation/stop-only). - RETIRED_WIRE_VARIANTS grows by ConsumeLifecycleLease @ wire v6 with the matrix extension (old-frame typed stale-wire-version refusal + audit). PROTOCOL_VERSION stays 6. - Telemetry op domain shrinks; nixos-modules/privileges-json.nix row deleted (drift gate green); all generated views regenerated (drift 19/19). - Tests: concurrent consume - exactly one winner; replay refused (one-time cell); restart-replay resistance proven through the durable store; envelope dispatch tests for both kernels; profile tests updated (retired op leaves host/guest catalogs). Gates: contracts 49/0, broker lib 684/4 (known env), matrix 7/7, profiles 4+2+5, d2bd 460/0, xtask 339+1 (known env), layout rc0, generated drift 19/19, nix drift green.
…backlog (U13) The check-async-gate probe (blocking calls in async contexts, KTD9) is now part of the enforcement chain: //bazel/checks/policy:check-async-gate (sh_test mirroring provider_crate_layout) is wired into the //bazel/checks :check suite, `make check-async-gate` aliases it, and `make check-ci` enforces it. The gate's 26-site pre-existing backlog is cleared: - Production sites (d2bd): interaction_composition, plane_port, process_provider_runtime (nix send), zone_enrollment, and the two composition.rs sites (storage-lifecycle report removal, guest state root mkdir) now run their blocking fs/socket calls through tokio::task::spawn_blocking with the same error mapping and cleanup ordering. - Test sites (zone_provider_acceptance, relay_service): async test fns that only performed blocking setup/teardown became sync tests; where a test genuinely awaits, the calls are wrapped in spawn_blocking. Gate result: `check-async-gate` scans 755 files, zero blocking calls in async contexts. d2bd lib 460/0, device-security-key tests green.
…first census family) Thirteen network-family operations (CreateTapFd, CreatePersistentTap, DeletePersistentTap, SetBridgePortFlags, CreateBridge, DeleteBridge, ApplyRoute, ApplyNftables, ApplyNftablesProjection, ApplyNmUnmanaged, ApplySysctl, UpdateHostsFile, SeedDnsmasqLease) retire their typed BrokerRequest/BrokerResponse wire variants at wire v6, mirroring the U10 process-family template: - Wire: variants + arms gone (13 dispatch arms, 660 lines); RETIRED_WIRE_VARIANTS grows to 24 entries @ v6; matrix extended (24 retired frames refused with the typed stale-wire-version refusal + audit; current-only frames are envelope invocations). PROTOCOL_VERSION stays 6. - Catalog: the 13 family rows stay committed (declaringProvider d2b-provider-network-local) with wireVariant dropped and provenance request; 13 broker-generic kernel rows (apply-nftables, apply-nftables-projection, apply-nm-unmanaged, apply-route, apply-sysctl, create-bridge, delete-bridge, create-persistent-tap, delete-persistent-tap, create-tap-fd, set-bridge-port- flags, update-hosts-file, seed-dnsmasq-lease) served in-broker by the kernel dispatcher, wrapping the existing ops/ modules + live helpers (97 rows total). CreateTapFd carries its fd through the envelope response (fd_indexes/kinds). - Sandwich: d2b-provider-network-local/src/operations.rs hosts the 13 family handlers (typed payload validation + nested kernel invocation through the KernelCaller seam with the KTD6 evidence chain via spawn_blocking); d2bd's NetworkEffectPort and the host-prep/destroy/reconcile DAG legs now call the kernels directly over the origination socket (dispatch_broker_kernel_ack + host_nft_kernel_payload), leaving zero typed network refs in d2bd. - Ratchet: kernel_ops.rs network tokens seeded (nftables/dnsmasq -> network-local, sysctl -> activation-nixos, retires_with U12 network-fds census step); stale runtime.rs dnsmasq row removed. - Tests: kernel dispatch suite for all 13 kernels (admit/refuse surfaces, field contracts, typed-payload refusals, create-tap-fd missing-intent path), 8 network provider handler tests (nested frames, refusal preservation, fd round-trip), profile tests migrated, matrix current-only frames rewritten. Also in this change (review-fix follow-ups on the committed U10/U11 surface): - LaunchPosture::validate_request_fds refuses the ControllerEscrow posture outright (the retired arm refused a controller spawn without its bootstrap escrow fd; the fd-less family row can never transport it - fail-closed restored). - The spawn-process kernel result no longer advertises extraFdIndexes (the kernel path mints no extra response fds; the handler keeps fd 0 only). - The lifecycle-leases cell key space bumped to lifecycle-leases-v2: the U11 caller keys the one-time cell on the camelCase payload while the pre-upgrade arm keyed the same cell name on a snake_case/Debug identity, which would make pre-upgrade consumed leases replayable across an upgrade; the renamed cell separates the key spaces by construction. - The three host-prep/destroy broker-unreachable tests carry real fixture bundle artifacts (U12 legs resolve trusted intents before dialing) and the destroy fake broker answers envelope kernel frames. Gates: contracts 49/0, broker lib 685/4 (known env), matrix 7/7, profiles 4+2+5, d2bd 460/0, network-local 44+8, provider-process 45/0, toolkit 16 (+1 known env), xtask 339+2 (+1 known env), layout rc0, async gate 757 files clean, generated drift 19/19, nix drift green.
The legacy `check-provider-layout` command (four provider-crate paths) was superseded by the U14 canonical `check-provider-crate-layout` gate (provider_crate_policy: crate layout + family-knowledge ratchet, wired into //bazel/checks:check as //bazel/checks/policy:provider_crate_layout). Remove the command, its implementation, and its fixture tests; the usage string now names only the canonical gate. The plan's stale `check-provider-layout` reference is thereby unwired repo-wide.
The first full //bazel/checks:check run on the post-U12 tree surfaced a cluster of gate issues, all fixed here: - broker_protocol_compatibility bazel target: srcs now include tests/common/mod.rs (the test declares `mod common;`) and the target deps the broker crate itself (the matrix reads RetiredWireVariant/retired_wire_variant) - the target never built under bazel on this branch until the full suite exercised it. - Source hygiene (tier0_first_pass): non-ASCII dashes normalized to ASCII across 11 source/script files (incl. the plan doc; formatting only). - Telemetry domain pins: telemetry_frame + redaction_guard fixtures named the retired SpawnRunner op label; re-pinned to the live EnvelopeInvoke row. - spawn_process stale-socket test: the bind path inherited the sandbox execroot and exceeded SUN_LEN; the stale socket now binds a short /tmp path. - d2bd bazel lib tests excluded from //bazel/checks:check (rust-main- packages) with a recorded follow-up: the U10 seam exposed a pre-existing dual-rlib test-support graph (d2b_core vs d2b_core_test_support through the volume/session/provider test-support chains) that makes d2bd_lib_test_support unbuildable; cargo covers the d2bd lib tests (460/0) and the test-support variant-graph unification is tracked. - d2b-provider-process bazel test-support target gains the d2b-contracts-broker + d2b-core deps its seam code needs. Remaining gate failures are environmental: the 6 flake-eval nix targets cannot fetch the bazel execroot as a git tree (the execroot contains symlinks; they need a real checkout) - unrelated to this branch's code. Layer-1 suite: 462 pass / 6 env-fail.
The standing `make check-clippy` gate was never exercised during the U10-U14 unit work (workers ran focused tests only), so the workspace's denied-lint backlog accumulated and blocked it; this clears it to GREEN (with the CARGO_BIN_EXE_d2b-broker env the env!()-based broker tests require): - d2b-contracts-broker: doc/attribute blank-line hygiene, iter().cloned() -> to_vec(), and the wire-contract constant pins carry a justified #[allow(clippy::assertions_on_constants)] (the literals are the shared WIRE codes, not tautology). - d2b-broker: doc-gap hygiene, collapsible ifs (let-chains), Option::map, contains over iter().any(), needless Option::as_deref, String conversion, elided lifetime, needless cast, return-let; the spawn-process stale-socket test binds a short /tmp path (SUN_LEN under the bazel sandbox). - d2b-provider-toolkit: too_many_arguments allows on the kernel-carrier surface (documented contract), needless Option::as_deref. - d2b-provider-process: too_many_arguments allow on the control-identity helper. - d2bd: doc-gap + doc-list hygiene, panic_message via match, Error::other, identity-map removal, and the effect-service poll timers are now tokio::spawn(ractor::time::send_after(...)) - a bare `let _` previously DROPPED the reschedule future (the poll loop would stop after the first interval). - xtask: collapsible ifs, async-gate scanner repeat_n/vec-fill, dependency- surface sync-tool gets a file-level disallowed_methods allow (the U13 deny list targets runtime workers, not this build-time audit). - d2b-broker-composition, d2b-provider-supervisor, d2b-provider-network-local tests: hygiene + small lints. Regression-checks: d2bd lib 460/0, broker lib 685/4 (known env), network-local 44+8, contracts/telemetry/toolkit/providers green, async gate still 0 violations (the timer spawn uses tokio, not the blocked calls list).
The live plan still named the removed `check-provider-layout` command in its Verification Contract and U14 bullet; re-point both to the canonical `check-provider-crate-layout`. Also dedent the async-gate scanner's for-loop body left over-indented by the let-chain collapse in 6176b28.
The retired family rows declare closed payload schemas (additionalProperties false) that do not carry tracingSpanId, but the typed request structs serialized tracing_span_id as `null` when None - so every supervisor/daemon envelope call carried an undeclared key and the real broker's envelope validation refused it with invalid-payload. Unit tests used fake brokers and never hit the real validation; the host-integration vmChecks (virtiofsd volume runtime, device-worker launch, guest preflight, state posture, resource-operator activation) caught it: the virtiofsd spawn failed with "kernel refused: invalid-payload". All 51 request structs now skip serializing an absent tracing_span_id (#[serde(default, skip_serializing_if = "Option::is_none")]); a present span still serializes normally. Also regenerate docs/reference/daemon-api.md (stale after the post-U12 commits - the gen_daemon_api_drift CI check).
The committed rows name family operations in PascalCase (ObserveRunner, ApplyNftables) while the family ResourceRef labels are lowercase and dash-separated (Operation/observe-runner, Operation/apply-nftables). The U12-era case-insensitive match never reconciled the dash, so the daemon's rendezvous refused every forwarded family call with uncommitted-operation (the host-integration vmChecks proved it: the supervisor's ObserveRunner forward was refused). declares() and the bare-name invoke lookup now canonicalize both spellings (case + dashes) before comparing.
…pe seam The host-integration vmChecks exposed a second U10 integration gap: the provider-controller launch path pre-arms a seqpacket bootstrap pair, but the family rows declared no fd facet, so the supervisor refused every inherited descriptor and the controllers could never launch (the known "daemon launch path passing escrow fds needs later migration" item). This restores the retired escrow contract through the seam: - The SpawnRunner family row declares the request-fd facet (maxFds 3, fdKind Any), so the envelope's U1 fd validation admits the escrow leg. - The supervisor's launch passes the ticket's inherited fds through the envelope request leg (envelope_call_with_fds) and reports the inherited count; the deep refusal is removed. - The spawn-process kernel pops the controller bootstrap fd (the daemon end), inserts it into the broker-side controller-bootstrap registry under the runner id, and returns a duplicate as the extra response fd (controllerBootstrapFdIndex + extraFdIndexes). - The family handler validates per posture again: ControllerEscrow requires its escrow descriptor (the retired (0,0) fail-closed), other postures refuse any descriptors; the forwarded request fds ride into the nested kernel call and the response's bootstrap index is forwarded. - The wire-opacity storage tests aligned with the span omission from 9464800 (an absent tracing span is no longer serialized). Broker lib 685/4 (known env), supervisor 8, process 45, d2bd 460.
The resolver-side inherited-fd fence (a U10-era placeholder for the fd-less family rows) refused every controller ticket before the launch; with the SpawnRunner row now declaring the request-fd facet, the fence is removed and the per-posture shape is enforced fail-closed in the family handler.
An outer forwarded invocation parks its dispatch worker on block_on until the nested leg's reply crosses back. One shared round-robin pool could land the nested leg on a pinned worker's queue and deadlock the pair until the io budgets expire - the SignalRunner/open-pidfd 10s stalls. Nested legs now dispatch on a dedicated pool, making the starvation impossible by construction.
…entity Two VolumeBinding workers share the template-scoped role id, so the daemon's (vm, role) pidfd table kept only the first registration and every later ObserveRunner resolved to the wrong pid - the readiness probe then quarantined the row and its retries livelocked against the broker's duplicate-spawn guard. Roles that admit several runners per VM (virtiofsd: one per microvm.shares row) now register and are looked up under '<role-id>@<resource-uid>'; single-runner roles keep the bare role id the vm-stop signal paths key on. SignalRunner carries the closed role on the wire so its lookup composes the same key.
Two VolumeBinding workers share the template-scoped role id, so the daemon's (vm, role) pidfd table kept only the first registration: the second worker's ObserveRunner resolved to the first worker's pid, its readiness probe failed the cgroup check, the row quarantined, and every retry livelocked against the broker's duplicate-spawn guard. Serving-worker launches (the binding-owned virtiofsd worker class) now register and are looked up under '<role-id>@<resource-uid>' - a key the supervisor derives from the resolved intent's serving-worker class and the family handlers derive from the same predicate (resolved_intent_is_serving_worker). Single-runner roles keep the bare role id the vm-stop signal paths key on. SignalRunner carries the bundle runner intent ref so its lookup resolves the same class.
… zone prepare_state_dir resolved the kernel zone from the Device's Guest target VM, but a nested Guest is never registered with the host daemon's zone coordinator - every reconcile failed with zone-coordinator-vm-not-registered and the swtpm worker could never launch. The Device row's own zone (the same zone the declared-children surface keys on) is the correct anchor.
…tory parent The host unit provisions /var/lib/d2b/tpm-state owned by the d2bd service account (the declared daemon-owned anchor the Device state subdirectories are created inside); the broker's production root-parent rule refused it and every TPM reconcile died on the prepare-directory kernel. The prepare-directory kernel now carries the broker's daemon uid and admits a parent owned by that trusted account; the vm-start prerequisite path keeps its exact prior posture.
…state dir is absent The declared swtpm Process row's driver launches as soon as the row exists; a launch racing the backing state Volume's layout spawned a child that died on its first log write and burned the row's restart budget before the layout ever created the directory. The launch now refuses retryably until the trusted state directory exists.
…nd supervisor probes
…ption The swtpm_ioctl -i flush is a one-shot: its clean exit is the success path. The launch's adoption read treated a child that already exited as a failed launch (pidfd-unavailable) and the non-retryable ephemeral row died before the reaper log could publish the exit outcome. A vanished EphemeralProcess spawn now reports the launch; the exit outcome itself decides the row.
The observation logs the broker-seam chase added (invocation timing, fd-count and cgroup probes, pidfd-table markers, executable-mismatch diffs) have served their purpose; the fixes they carried stand on their own. Error-path diagnostics on code introduced by the fixes stay.
The U6 composition split left d2b-broker's integration tests compiling against a CARGO_BIN_EXE env var cargo never sets for the package (the bin moved to d2b-broker-composition); the harnesses now resolve the binary at runtime from the cargo var or the workspace build output. The test constructors gained the fields the nested-dispatch pool, the TPM zone anchor, and the prepare-directory daemon-uid threading added.
# Conflicts: # packages/d2b-broker/BUILD.bazel # packages/d2b-core/BUILD.bazel
The v3 merge (#539) folds a per-crate clippy aspect into every suite member and the workspace `-Dwarnings` flag carries into the clippy action, so the warnings the merge introduced are gate failures now: * provider-supervisor: a needless borrow on the resolved intent, a collapsible `if let` chain over the launched-runner snapshot, and a `map_err` that only logs before returning the same error (`inspect_err`). * provider-process: the launched-runner snapshot tuple trips `clippy::type_complexity`; the tuple is the trait's wire shape, so the allowance names that instead of aliasing it. * d2b-broker kernel ops registrations pass the kernels as closures; the context now takes the function paths directly. * d2b-broker's BUILD keeps the U6 split (no binaries) but must still export the `:clippy` aggregate `//bazel/checks/rust:portable_rust_broker` names. * `maybe_harden_swtpm_dir` matched `BrokerRequest::EnvelopeInvoke` on `envelope.request`, which is a `BootstrapCall` under `layer1-bootstrap`: no bootstrap leg can present a chain root, so that variant pins the nested-pool decision to false. * the broker test harness resolved the binary through `env!( "CARGO_MANIFEST_DIR")`, which embeds the absolute sandbox path the bazel rustc wrapper refuses; the fallback resolves the workspace target dir from the test's runtime cwd instead.
…emon The zone-native fallback restored in fe6e17f composed the Device state Volume directory (`device-<32hex>-tpm-state`) locally, which the shared-crate family-knowledge probes read as per-resource naming knowledge in `d2bd` and which the L1 policy gate refuses (the ratchet only shrinks). The broker's `PrepareStateDir` resolves that same directory from the same trusted artifacts - the request's `baseDir` is not read on either arm - so the daemon now sends the trusted `path:swtpm-state:<guest>` row's root and leaves the narrowing where the owning artifacts live. `state_volume_name`, `unique_tpm_state_dir`, `tpm_devices_of_guest` and `deterministic_resource_uid` had no other caller and are gone with it.
… flush db6ff2d put the presence check inside `derive_resource_backed_paths`, which made the derivation a filesystem probe: the identity fence then lost its precedence (a foreign state dir reported "not provisioned" instead of the identity mismatch) and the pure path-derivation fixtures could no longer accept a well-formed plan, while the check read the state *root* rather than the Volume directory the child opens. Presence now lives in the spawn hook that actually proceeds with the launch, over `trusted_state_dir` - the state root joined with the trusted Volume name the argv fence already pins - and only for the long-lived worker, the launch whose argv names `--tpmstate dir=`: `swtpm` opens its log and pid file there the moment it starts, so a launch racing the Volume's layout must fail retryably instead of burning the row's restart budget on a child that dies on its first write. The one-shot `swtpm-init-flush` is admitted instead: its argv names `--unix <dir>/ctrl.sock`, a control socket the worker binds inside that same directory, so the flush waits for the socket and succeeds once the worker is up - refusing it would spend the row's single launch attempt (one-shots never restart) on a race it can win. The device-worker-launch vmCheck failed exactly this way: the daemon admits the declared flush row at startup, before the Volume layout lands. Also pins the delete-TAP payload to the six identity/generation keys it carries: 9464800 omits the absent tracing span from envelope payloads, which the count assertion had not followed.
…orary)" This reverts commit c1ced5e.
`runtime::tests::a_current_wire_variant_passes_the_retired_wire_gate` sends a current Hello over a socketpair, so the peer-credential gate reads the kernel peer uid of the test process and compares it with the fixture's `config.d2bd_uid`, which was pinned to 1000. That only holds on a host whose runner is uid 1000: a GitHub runner is 1001, where the gate refuses the request (`Broker.PeerCredentialRefused`) and the three broker lib-test targets fail in CI while passing locally. The fixture now names the test process's own uid - the same `Uid::current()` the production config falls back to - so the gate compares like with like on any runner.
…orary)" This reverts commit a6b6e8a.
This was referenced Sep 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Completes the broker-seam refactor (
docs/plans/2026-09-14-001-refactor-provider-services-broker-seam-plan.md, U1-U14). The typed process-family wire variants retire behind the genericEnvelopeInvokecarrier; the privileged cores run in-broker as broker-generic kernels; family handlers validate daemon-side and graft the U9 evidence chain (KTD6).Fixes #523 · Related #516
Units
d2b-provider-process, supervisor envelope migration, production chain-audit sink (write_chain_record), P1 spawn-kernel ports (usbip binds / serving-worker ACLs / stale-socket cleanups / duplicate-spawn guard).consume-cell/complete-cellkernels (row+arm deleted; AE2 restart-replay; one-winner).d2b-provider-network-local, daemon legs viaenvelope_invoke_kernel.check-async-gatewired into//bazel/checks:check+make check-ci; 26-site blocking-call backlog cleared.provider_crate_layoutpolicy gate (family knowledge ratchet, generated-only classification); stalecheck-provider-layoutremoved.Gates (local)
make check-clippygreen;//bazel/checks:check462 pass (6 env-onlyflake-evalsandbox failures); generated-artifact drift 19/19; nix privileges drift PASS; tier0 hygiene PASS; matrix 7/7 (retired frames -> typedstale-wire-versionrefusal + audit, never silent drops); conformance 13/13; d2bd lib 460/0; broker lib 685/4 (4 proven-pre-existing env failures); cross-binary//tests/unit/gates:broker-seam-pilotgate wired (run requires root for the production bundle posture).make test-host-integration(vmChecks, KVM) — pending local run; CI status.