daemon-0.2.2: trace flag, XDP mode + structured attach log, distinct link-add codes

[KRuskowski]

Daemon-side changes from the 0.2.2 brief from the benchmark agent. Each P0 item is one commit; all four pass local repro.

P0.1 — diagnostic surface for t1-integrity (8a13935)

• New --trace-forward-hashes daemon flag (off by default). When enabled, every wg-relay forward logs a SHA-256(payload) hash + length + source/destination peer pubkey prefix at two points: ingress (after the source peer matches and after optional MAC1 verification) and egress (just before the sendto to the destination). The same SHA on both lines proves the relay didn't mutate the frame; divergence pinpoints a corrupting code path. Per-frame log on the hot path — start warns the operator and the docs note it must not be left on in production.
• wg peer list now surfaces per-peer drop counters peer.<i>.drop_no_link and peer.<i>.drop_pubkey_mismatch. These are the pair-attributable subset of the existing aggregate counters. Aggregate counters are unchanged; the other drop classes (drop_unknown_src, drop_not_wg_shaped, drop_handshake_no_pubkey_match) are by definition unattributable to a known peer and stay aggregate-only.

Operator-observable behaviour: with the runner patched to pass --trace-forward-hashes for the integrity test specifically, the relay logs ingress + egress SHA pairs the runner can compare end-to-end. wg peer list shows where each known peer's drops are landing without diffing aggregate counters between bursts.

P0.2 — XDP attach (acb29ec)

• Structured xdp_attached iface=<nic> ifindex=N mode={drv,skb} driver=<drv> kernel=<release> log line per attached NIC.
• New --xdp-mode={drv,skb,auto,off} flag, default drv. The previous code unconditionally tried DRV then fell back to SKB — the silent fallback that on cloud-gcp-c4 labelled rows "xdp" that were really running on the userspace recv loop. auto preserves the historical behaviour explicitly; off skips XDP attach even when --xdp-interface is set.
• Attach failure is FATAL. XdpAttach failure logs xdp_attach_failed iface=<nic> ifindex=N mode=<mode> driver=<drv> kernel=<release> reason=<errno-string> and WgRelayStart returns nullptr → main.cc exits non-zero. The runner sees a clear failure signal rather than a silently-degraded relay.
• gVNIC / vmxnet3 / virtio-net advertise XDP only in generic mode on stock kernels through 6.12 — operators on those NICs need --xdp-mode=auto (or =skb) to opt in to the generic fallback explicitly. Documented in --help and in the 0.2.2 CHANGELOG note.

Operator-observable behaviour: a fresh daemon launch on the cloud fleet either attaches XDP and logs the achieved mode, or fails fast with the structured xdp_attach_failed line. Userspace path is no longer reached implicitly.

P0.3 — link-per-peer limit (8137a2c)

Decision: option (b) from the brief. The "each peer is in at most one link" invariant is load-bearing — it's the iter-1 design contract that lets the relay forward purely on the source 4-tuple. Multi-link mesh routing is future work (per-link UDP ports or in-packet introspection); not on the path for 0.2.2.

• New WgRelayLinkAddResult enum (kWgLinkOk / UnknownPeer / SelfLink / LimitExceeded / Duplicate) plus WgRelayLinkAddDetail. Bool WgRelayLinkAdd kept as a thin wrapper for callers that only need success/failure.
• WgLinkAdd einheit handler maps each non-OK result onto a distinct error code: wg_peer_unknown, wg_link_self, link_limit_exceeded, wg_link_duplicate. The runner now has a discriminator to switch on; star-topology configs that hit the limit can fall back to disjoint pairs without parsing free-form error text.
• wg link add verb description in the registry calls out the iter-1 limit so wg link add --help documents it.

Operator-observable behaviour: a wg link add that exceeds the one-link-per-peer limit returns error.code = link_limit_exceeded with a one-line message explaining the rule. Other failure modes (typo'd peer, duplicate pair, self-link) return distinct codes too.

P1 — verb registration (verified, no code change)

einheit_channel.cc:2191-2254 registers all 11 wg_* verbs (peer add/pubkey/nic/remove/list, link add/remove/list, show, show config, blocklist list) with handler, role, dotted path, description, and arg schema. The round-1 oneshot: no matching command was the einheit CLI's hd_relay adapter (separate repo, out of scope).

Test coverage

• tests/test_wg_relay_trace.cc (5 cases): trace on/off, invalid --xdp-mode, --xdp-mode=off, per-peer drop counter increment.
• tests/test_wg_relay_links.cc (7 cases): all five outcome codes, the brief's star topology, the runner's expected fallback (disjoint pairs).

12/12 local pass.

Out of scope (confirmed)

No changes in ~/dev/HD.Benchmark, ~/dev/einheit, DERP mode, HD-Protocol mode, or the wg-relay steady-state forwarding hot path. Trace logging is gated behind a single bool check; per-peer counter bumps sit next to existing aggregate increments. The 0.2.1 24 h T2 soak signal stands.

[KRuskowski]

Thanks for the review — addressing each point.

1. P0.3 — option (b) rationale, with file:line

The iter-1 limit is load-bearing in the data plane, not incidental scaffolding:

• bpf/wg_relay.bpf.c:43-59 — the XDP ep_val is a single egress endpoint per source 4-tuple (one IP + one port + one ifindex). Not a list, not a tagged variant.
• bpf/wg_relay.bpf.c:74-79 — the wg_peers BPF map is BPF_MAP_TYPE_HASH with key=ep_key (src 4-tuple) → value=ep_val (single partner endpoint). Multiple links per peer would have multiple values for the same source key — the map type doesn't allow it.
• bpf/wg_relay.bpf.c:307-309 — the XDP fast path looks up exactly one partner: bpf_map_lookup_elem(&wg_peers, &src_key) returns one ep_val and forwards there. There's no disambiguation logic; with multiple partners possible, the kernel would silently arbitrarily pick one.
• src/wg_relay.cc:710-724 — userspace FindLinkPartnerLocked mirrors the same "first match wins" behaviour by linear-scanning r->links. With multi-link peers it'd silently pick whichever link is first in the vector.

Relaxing to N≥4 (option a) would require: (1) BPF map type change to a list-valued map or a different key scheme, (2) some way to disambiguate the destination from the packet — the relay has nothing other than the source 4-tuple to go on, since it deliberately ignores WG protocol contents. That's a multi-week design change for iteration 2. Option (b) — distinct error code so the runner sees the rejection cleanly — is the right call for 0.2.2.

The iter-1 invariant is documented in docs/design/wg_relay_design.md and in the project memory; I should have cited it in the commit message originally.

2. P0.1 — counter set was incomplete; fixed in 7b3deb0

You're right. drop_handshake_no_pubkey_match always has a source IP available, even though it isn't attributable to a known peer, and it's specifically the failure mode the c4_gcp internal/external NAT bug produces.

Added a per-source-IP histogram in 7b3deb0 covering the three classes that fall outside per-peer attribution: drop_unknown_src, drop_not_wg_shaped (userspace path), and drop_handshake_no_pubkey_match. Surfaced via a new wg show drop_sources verb that emits one row per source IP with the three counters + last-seen timestamp. Bounded at 256 IPs with FIFO eviction so a spoofed-source flood can't grow the map without bound.

XDP-path drop_not_wg_shaped stays aggregate-only — populating a per-source map from BPF would need new map plumbing and isn't on the diagnostic path the brief flagged. The userspace shape filter (which runs when XDP isn't attached or when packets XDP_PASS for shape reasons) is covered.

3. P0.1 — t1-integrity reproduction status

P0.1 ships diagnostics. Integrity reproduction is blocked on the runner agent enabling --trace-forward-hashes for the integrity test specifically.

The trace flag's own smoke tests pass (the 5 test_wg_relay_trace cases), proving the new code paths are live. I did not run the integrity test against a 0.2.1-equivalent build with trace on — that requires the runner-side change to flip the flag for that test, which the brief explicitly puts out of scope ("HD.Benchmark/tooling — different agent, different repo").

Recommend that as the next runner-side change. Once the runner enables --trace-forward-hashes (and now optionally wg show drop_sources for the per-source breakdown), the daemon-side diagnostic surface is sufficient to triage what's currently a one-log-line failure.

4. P0.1 — flake fix nature: test-only, no daemon change

Both fixes in b4b9b76 are test-side. No production race.

Port-bind retry: the test harness binds a UDP socket on port 0, reads the assigned ephemeral port, closes the fd, and passes the port to WgRelayStart to rebind. There's a TOCTOU window between close and rebind where another process can grab the port. Pure harness issue; fixed by retrying SetUp up to 5 times with fresh ephemeral ports.

Counter-poll: the test calls recv() on bob's socket then immediately reads WgRelayGetStats. The relay loop's order is sendto → trace egress log → fwd_packets.fetch_add, and sendto is what makes the kernel deliver bytes to bob. So recv() can return in the test while the relay thread is still microseconds away from fetch_add on the counter — read it too soon and fwd_packets is still 0 even though the packet was forwarded. This is a benign read-after-write skew, not a correctness issue: in production wg show is polled on operator timescale and the lag is invisible. Fixed by polling the counter up to ~500 ms before asserting.

The trace-log line firing without the counter incrementing is the same skew window — the egress log is emitted between sendto and fetch_add. That ordering is intentional (we want to log "we sent it" before claiming the forward succeeded in counters); the test just needs to acknowledge the window.