[softnpu, mcast] wire up the softnpu backend for multicast table ops by zeeshanlakhani · Pull Request #405 · oxidecomputer/dendrite

zeeshanlakhani · 2026-10-15T13:37:09Z

Previously, SoftNPU didn't interact with multicast. mc_group_create and mc_port_add returned a stubbed OperationUnsupported, and port remove/destroy were no-ops.

With the merge of bitmap-based replication in oxidecomputer/p4rs#240, SoftNPU can now replicate off of port bitmaps carried in the replication table entry itself. On Tofino, membership subscriptions live in the PRE and change independently of match-action state. On SoftNPU, membership is captured at write time, and a membership mutation involves rewriting these affected replication entries.

Includes:

asic/SoftNPU:
- Live replication entries are tracked per group and rewritten when members are added/removed. The replication tracking lock covers both the snapshot and table updates.
- Source filter, mcast NAT (dst-only, per [multicast] Bitmap-based multicast replication for sidecar-lite sidecar-lite#152), egress decap, and mcast src MAC tables now map onto their sidecar-lite tables.
  - egress.asic_id_to_port maps ASIC port IDs to logical port numbers for indexing the decap bitmap (same as on Tofino).
- TableRemove is sent before TableAdd because p4rs drops an add whose key already exists and keeps the older entry (action and params).
dpd/mcast:
- mcast::reset deletes each known group's entries before the table clears. SoftNPU's Table::clear is still a stub.
- del_entries_where skips the stale-entry table sweep when the backend cannot list entries (e.g., in SoftNPU).
deps
- softnpu-lib + p4rs have been updated on main ([multicast] softnpu with bitmap-based replication softnpu#183, [multicast] Bitmap-based replication and bit-slice assignment for softnpu/a4x2 p4#240). SoftNPU must still loads [multicast] Bitmap-based multicast replication for sidecar-lite sidecar-lite#152 at runtime where used (until its merged).

Previously, SoftNPU didn't interact with multicast. `mc_group_create` and `mc_port_add` returned a stubbed `OperationUnsupported`, and port remove/destroy were no-ops. With bitmap-based replication in p4rs ([#240](oxidecomputer/p4#240)), SoftNPU can replicate off of port bitmaps carried in the replication table entry itself. On the Tofino, membership subscriptions live in the PRE and change independently of match-action state. In the SoftNPU work, membership is captured at write time, and a membership mutation means rewriting the affected replication entries (when triggered). Includes: - *asic/SoftNPU*: - Live replication entries are tracked per group and rewritten when members are added/removed. The replication tracking lock covers both the snapshot and table updates. - Source filter, mcast NAT (tagged and untagged), egress decap, and mcast src MAC tables now map onto their sidecar-lite tables respectively. - `egress.asic_id_to_port` maps ASIC port IDs to logical port numbers for indexing the decap bitmap (same as on Tofino). - `TableRemove` is sent before `TableAdd` because p4rs drops an add whose key already exists and keeps the older entry (action and params). - *dpd/mcast*: - `mcast::reset` deletes each known group's entries before the table clears. SoftNPU's `Table::clear` is still a stub. - `del_entries_where` skips the stale-entry table sweep when the backend cannot list entries (e.g., in SoftNPU). - softnpu-lib is pinned to `zl/multicast` @ softnpu#183, which pulls in p4#240. SoftNPU loads sidecar-lite#152 at runtime.

This adds multicast replication to sidecar-lite, emulating DPD's sidecar.p4 on dendrite up through oxidecomputer/dendrite#405 following RFD 488. We manage 2 bitmaps per underlay group (external and underlay), for which replication is driven by p4rs `replicate` (oxidecomputer/p4#240; we track p4 on main now). Ingress zeroes out at most one of these bitmaps determined by the geneve multicast option tag (`tag=0` zeroes out underlay; `tag=1` zeroes out external), and merges both. Tag 0 (External) also decaps upon ingress. With no tag given, as on external NAT ingress packets, both bitmaps remain in place. Only geneve-encapsulated multicast is replicated. Multicast packets that miss ingress NAT drop on the empty bitmap, matching DPD's `drop_mcastv{4,6}_no_group` semantics. Packet copies that remain encapsulated lose a hop on the outer IPv6 header, as modelled in `MulticastRouter6`. External multicast is NAT-encapsulated into the underlay via `nat_v{4,6}_mcast(_untagged)`, reusing `forward_to_sled`. Tagged and untagged variants exist as separate tables since x4c cannot key directly on header validity. The encap piece now zeroes out the outer src, traffic class, flow label, and inner eth src, like DPD's `add_encap_headers`. Egress decap checks `ff04::/64` and `tag=2`, matching DPD's egress `mcast_tag_check`, with per-port decap from a RID-keyed bitmap (`tbl_decap_ports`) and optional VLAN. `tag=2` decapsulation also rejects inner unicast. `tag=0` copies (already decapped in ingress) use the table for the VLAN. Dst MAC derivation (RFC 1112, RFC 2464) and per-port src MAC rewriting run on every packet copy. Includes: - The parser rejects `ff01` addresses outright (`ff02` *does* still go to the scrimlet). - Multicast packets with a TTL/hop_limit <= 1 get dropped in ingress after the `fwd_from_scrimlet` return, letting scrimlet transmissions pass through. - Both decapsulation paths drop inner TTL <= 1. IPv4 checksums use RFC 1624 (eqn. 3): decrementing the TTL adds 0xfeff in one's complement, which yields 0x0000 rather than 0xffff at the zero boundary, matching a full recompute (XOR is used here due to `~` not being parsable by x4c). - scadm gets add/remove/dump for replication (v6), source filters, mcast NAT, decap ports, src MAC, and asic-id-to-port (for logical mapping). - Unicast modifications: - The router drops TTL/hop_limit <= 1 and decrements remaining packets on routes to nonzero ports, updating the IPv4 checksum. - Port 0 (the scrimlet) skips the `<= 1` check and the decrement, while a TTL/hop_limit `= 0` still drops before route lookup, except on the boundary services decapsulation path. - DPD's Router4/6 still decrement service-bound packets. - The NAT encap zeroing described above is shared with unicast NAT. - Packets with a parsed Geneve header skip ingress NAT, including unicast.

cfzimmerman

Voxel exercises softnpu, right? Can you run a voxel test and post some validation results? I'll stamp once we know it works.

zeeshanlakhani · 2026-10-29T07:38:06Z

Voxel exercises softnpu, right? Can you run a voxel test and post some validation results? I'll stamp once we know it works.

Incoming with this line of work. Via oxidecomputer/voxel#75 and past iterations, I demo'ed the prototype set of branches working.

cfzimmerman

Left a reply on the table.rs diff. Otherwise looks good to me once we're confident in e2e voxel testing.

This also replaces ``make_tag(TEST_TAG)` w/ `TEST_TAG`` in two dpd-client mcast tests.

rcgoodfellow · 2026-11-06T18:55:21Z

+        // get removed here, one `TableRemove` per keyset, and every other
+        // table retains its entries.
+        //
+        // Send a `TableClear` once softnpu and propolis support one (the


This has landed, so looks like this can change now?

Yep, and I've pushed oxidecomputer/softnpu#234 to include a message for it. Definitely makes life easier.

Of note, the next push's update contains a clear on each table when dpd constructs that table at startup time. This is due to the fact that the softnpu pipeline outlives the dpd process. A restarted dpd then starts empty and the control plane reconciles what's missing.

rcgoodfellow · 2026-11-06T19:25:01Z

+/// Write an entry, or replace any entry already associated with the key.
+///
+/// Note: SoftNPU's `TableAdd` retrains an existing entry in place and reports
+/// nothing back. The backend key lookup does not return `AsicError::Exists`,


I think the bug here is in p4 codegen. The add entry codegen should use HashSet::replace instead of HashSet::insert as replacement independent of whether or not a matching key already exists in the table is the predominant match-action use case.

Yeah, I was holding off, but I'll just do this.

oxidecomputer/p4#282 @rcgoodfellow.

On a PUT to an mcast group awaiting repair, we now rewrite the egress decap bitmap from its stored membership before modifying membership (in the case that rollback may have dropped it). Underlay cleanup now deletes the replication entry first and then maintains the decap bitmap if the delete fails. Decapsulation holds a superset of all the ports replication can still send to.

Note on *deps*: - softnpu -> zl/table-clear (oxidecomputer/softnpu#234) - p4 -> zl/upsert-table-entries (oxidecomputer/p4#282)

@rcgoodfellow

…282) `HashSet::insert` retained the prev entry, which meant that adding the same key with a new action or new params was a no-op. We now use replace per @rcgoodfellow @ oxidecomputer/dendrite#405.

zeeshanlakhani added this pull request to stack #401 October 15, 2026 13:38

zeeshanlakhani self-assigned this Oct 15, 2026

zeeshanlakhani added the multicast Multicast-related feature/impl label Oct 15, 2026

zeeshanlakhani added 7 commits October 17, 2026 06:29

[merge] merge fwd

776db38

[deps] p4(rs) update

8c742a3

[merge] merge fwd

94ef914

[deps] softnpu and p4(rs) to main

b52e768

[merge] merge fwd

5671a7e

[merge] merge fwd -> drop VLAN dispatch for mcast NAT

90eacb1

[merge] merge fwd

4a52d14

zeeshanlakhani removed this pull request from stack #401 October 21, 2026 01:55

zeeshanlakhani added this pull request to stack #410 October 21, 2026 01:59

zeeshanlakhani requested review from FelixMcFelix, cfzimmerman, rcgoodfellow and taspelund October 21, 2026 02:37

[merge] merge fwd

d664792

cfzimmerman reviewed Oct 28, 2026

View reviewed changes

[merge] merge fwd

61c965a

zeeshanlakhani added 2 commits October 30, 2026 00:13

[review] address review items

1188ce3

[merge] merge fwd

05b4530

zeeshanlakhani requested a review from cfzimmerman October 30, 2026 05:45

zeeshanlakhani added 2 commits October 30, 2026 12:01

[merge] merge fwd

b69f67e

[merge] merge fwd

53f18f3

cfzimmerman approved these changes Oct 30, 2026

View reviewed changes

Comment thread asic/src/softnpu/table.rs Resolved

[review, softnpu] replace entry on add and don't drop on a dup add

563cb14

zeeshanlakhani mentioned this pull request Oct 31, 2026

[mcast, pt1] multicast address, source, and route key types oxidecomputer/maghemite#943

Open

[merge] merge fwd

3019e6f

zeeshanlakhani force-pushed the zl/mcast-softnpu branch from 70ff3bc to 3019e6f Compare November 2, 2026 16:34

zeeshanlakhani added 4 commits November 5, 2026 16:53

[merge] merge fwd

ef57482

[merge] merge-fwd

628e849

[softnpu, mcast] clear known src filters on update/delete + sweep after

eb55743

This also replaces ``make_tag(TEST_TAG)` w/ `TEST_TAG`` in two dpd-client mcast tests.

[merge] merge fwd

50c7013

zeeshanlakhani force-pushed the zl/mcast-softnpu branch from 3562104 to 50c7013 Compare November 6, 2026 06:11

rcgoodfellow reviewed Nov 6, 2026

View reviewed changes

This was referenced Nov 7, 2026

[codegen, fix] Replace (not insert) table entries with matching keys oxidecomputer/p4#282

Merged

[ddm, mg-lower, pt6] underlay membership sweep, admin v5, & mrib origination oxidecomputer/maghemite#999

Open

zeeshanlakhani added 2 commits November 9, 2026 04:56

[review] address feedback + table upserts & clears

c6392f8

Note on *deps*: - softnpu -> zl/table-clear (oxidecomputer/softnpu#234) - p4 -> zl/upsert-table-entries (oxidecomputer/p4#282)

zeeshanlakhani requested a review from rcgoodfellow November 9, 2026 05:07

zeeshanlakhani added 2 commits November 9, 2026 08:17

[deps] newly built softnpu

f885505

[merge] merge fwd

768f2c2

zeeshanlakhani removed this pull request from stack #410 November 10, 2026 16:05

zeeshanlakhani added this pull request to stack #420 November 10, 2026 16:06

Conversation

zeeshanlakhani commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cfzimmerman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zeeshanlakhani commented Oct 29, 2026

Uh oh!

cfzimmerman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rcgoodfellow Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

zeeshanlakhani Nov 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rcgoodfellow Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

zeeshanlakhani Nov 7, 2026

Choose a reason for hiding this comment

Uh oh!

zeeshanlakhani Nov 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zeeshanlakhani commented Oct 15, 2026 •

edited

Loading