Skip to content

p4: implement SC_ARP_NEEDED / SC_NEIGHBOR_NEEDED #181

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
taspelund wants to merge 11 commits into
base: main
Choose a base branch
from
+609 −177
Open

p4: implement SC_ARP_NEEDED / SC_NEIGHBOR_NEEDED #181

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
0bf3081
p4: implement SC_ARP_NEEDED / SC_NEIGHBOR_NEEDED
taspelund Oct 30, 2026
8c75174
[multicast] restrict scrimlet multicast to IPv6 link-local
zeeshanlakhani Nov 4, 2026
c174b21
Merge pull request #1 from oxidecomputer/zl/arp-needed-mcast
taspelund Nov 4, 2026
917fb1b
[test] update tests
zeeshanlakhani Nov 5, 2026
e0d5b8d
[nit] fix
zeeshanlakhani Nov 5, 2026
5885940
Merge pull request #2 from oxidecomputer/zl/arp-needed-mcast-test-fix
taspelund Nov 5, 2026
7e446ef
[nit] fix
zeeshanlakhani Nov 5, 2026
82bf148
[fmt]
zeeshanlakhani Nov 5, 2026
a3bb85b
softnpu: update scrimlet_mcast_copies test comment
taspelund Nov 5, 2026
c9dd501
[mcast] switch-zone mcast goes out the mcast path
zeeshanlakhani Nov 10, 2026
840ce97
[mcast, mac] switch-zone traffic out sc_egress; SMAC rewrite @ end of…
zeeshanlakhani Nov 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
323 changes: 183 additions & 140 deletions p4/sidecar-lite.p4
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,168 +27,186 @@ control ingress(
ipv4_ttl_decrement() ttl_dec;

apply {
//
// Check if this is a packet coming from the scrimlet. If so just
// forward it out the source port indicated by the sidecar header.
//
if (hdr.sidecar.isValid()) { fwd_from_scrimlet(); return; }

@rcgoodfellow rcgoodfellow Nov 10, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's still not clear to me why this logic should change. If a packet is coming from the switch zone, it should just be sent out the port the driver on the host identifies in the sidecar header. There is no resolution to do and any multicast should be handled within the host itself. e.g. we are acting more as a gigantic NIC in this case than a switch.

@zeeshanlakhani zeeshanlakhani Nov 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After some discussion with @rcgoodfellow, including:

yeah i'm fine with rewrite being at the end of the pipeline and happening on every packet except for those with multcast macs - that makes sense. What I was taking issue with was the seemingly very contrived logic on mac rewrites at the head of the pipeline from userland.

I'm going with the giant-NIC model. The next push will return the switch-zone path back to main's fwd_from_scrimlet (i.e., no replication/TTL/DMAC handling for anything coming from the switch zone itself). Instead, we'll move the SMAC rewrite to the end of the ingress control for non-mcast packets that make it through to that point and aren't dropped or headed to the scrimlet, akin to DPD's current sidecar approach. tfportd's re-injected packets get their SMAC applied there too, which is how sidecar.p4 already handles them.


///
/// Check if we need to proxy arp this packet.
///
if (hdr.arp.isValid()) {
bool proxied = false;
pxarp.apply(hdr, ingress, egress, proxied);
if (proxied) { return; }
}

bool mcast_dst = false;
bool is_mcast = false;
// TODO: Combine these checks when x4c supports && and ||.
if (hdr.ipv6.isValid()) {
if (hdr.ipv6.dst[127:120] == 8w0xff) { mcast_dst = true; }
if (hdr.ipv6.dst[127:120] == 8w0xff) { is_mcast = true; }
}
if (hdr.ipv4.isValid()) {
if (hdr.ipv4.dst[31:28] == 4w0xe) { mcast_dst = true; }
}

if (mcast_dst) {
if (hdr.ipv4.isValid()) {
// 01:00:5e + zero bit, then the low 23 bits of the
// group (RFC 1112 section 6.4).
if (hdr.ethernet.dst[47:23] != 25w0x0200bc) {
egress.drop = true;
return;
}
if (hdr.ethernet.dst[22:0] != hdr.ipv4.dst[22:0]) {
egress.drop = true;
return;
}
}

if (hdr.ipv6.isValid()) {
if (hdr.ethernet.dst[47:32] != 16w0x3333) {
egress.drop = true;
return;
}
if (hdr.ethernet.dst[31:0] != hdr.ipv6.dst[31:0]) {
egress.drop = true;
return;
}
}
if (hdr.ipv4.dst[31:28] == 4w0xe) { is_mcast = true; }
}

//
// If the packet has a local destination, create the sidecar header and
// send it to the scrimlet.
// Check if this is a packet coming from the scrimlet. If so, just
// forward it out the source port indicated by the sidecar header.
//
bool local_dst = false;
local.apply(ingress, hdr, local_dst);
if (local_dst) {
// If this is boundary services packet, it should have a geneve
// header. Decap and attempt to route.
if (hdr.geneve.isValid()) {
ingress.nat_egress_hit = true;
decap_geneve();
}
// If there is no geneve header this is a packet from outside the
// rack and should be sent to the scrimlet.
else { fwd_to_scrimlet(); return; }
if (hdr.sidecar.isValid()) {
fwd_from_scrimlet();
} else {
// Multicast with TTL/hop_limit <= 1 must not be forwarded
// (RFC 1112, RFC 8200 section 3). The check lives here, not in
// the parser, because scrimlet transmissions return earlier
// through fwd_from_scrimlet and must not be dropped.
if (mcast_dst) {
///
/// Check if we need to proxy arp this packet.
///
if (hdr.arp.isValid()) {
bool proxied = false;
pxarp.apply(hdr, ingress, egress, proxied);
if (proxied) { return; }
}

if (is_mcast) {
if (hdr.ipv4.isValid()) {
if (hdr.ipv4.ttl <= 8w1) { egress.drop = true; return; }
// 01:00:5e + zero bit, then the low 23 bits of the
// group (RFC 1112 section 6.4).
if (hdr.ethernet.dst[47:23] != 25w0x0200bc) {
egress.drop = true;
return;
}
if (hdr.ethernet.dst[22:0] != hdr.ipv4.dst[22:0]) {
egress.drop = true;
return;
}
}

if (hdr.ipv6.isValid()) {
if (hdr.ipv6.hop_limit <= 8w1) {
if (hdr.ethernet.dst[47:32] != 16w0x3333) {
egress.drop = true;
return;
}
if (hdr.ethernet.dst[31:0] != hdr.ipv6.dst[31:0]) {
egress.drop = true;
return;
}
}
} else {
attached.apply(ingress, hdr);
}
if (hdr.geneve.isValid() == false) {
nat.apply(hdr, ingress, egress); // check for ingress nat
}
}

//
// After local and NAT processing, basic packet forwarding happens.
//
// Unicast and multicast are bifurcated, mirroring dendrite's
// sidecar.p4 (the meta.is_mcast metadata check). Multicast traffic
// is replicated and bypasses the unicast nexthop resolver,
// whose default_action is drop. Each replica's destination and
// source MACs are derived in the egress control instead.
//
// IPv6 link-local multicast (ff02::/16) never reaches this point; it is
// marked local and handled by the earlier local path. The parser
// rejects interface-local scope multicast, while expired multicast
// (TTL/hop_limit <= 1) gets dropped on the non-local path above.
//
// IPv4 local network control (224.0.0.0/24, RFC 5771 section 4) is
// not dropped by address. The TTL check covers IGMP, OSPF and
// PIM, but senders such as mDNS and VRRP can use TTL 255.
bool is_mcast = false;
if (hdr.ipv6.isValid()) {
if (hdr.ipv6.dst[127:120] == 8w0xff) { is_mcast = true; }
}
if (hdr.ipv4.isValid()) {
if (hdr.ipv4.dst[31:28] == 4w0xe) { is_mcast = true; }
}
//
// If the packet has a local destination, create the sidecar header and
// send it to the scrimlet.
//
bool local_dst = false;
local.apply(ingress, hdr, local_dst);
if (local_dst) {
// If this is boundary services packet, it should have a geneve
// header. Decap and attempt to route.
if (hdr.geneve.isValid()) {
ingress.nat_egress_hit = true;
decap_geneve();
}
// If there is no geneve header this is a packet from outside the
// rack and should be sent to the scrimlet.
else { fwd_to_scrimlet(); return; }
} else {
// Multicast with TTL/hop_limit <= 1 must not be forwarded
// (RFC 1112, RFC 8200 section 3). The check lives here, not in
// the parser, because scrimlet transmissions take the
// fwd_from_scrimlet branch and must not be dropped.
if (is_mcast) {
if (hdr.ipv4.isValid()) {
if (hdr.ipv4.ttl <= 8w1) { egress.drop = true; return; }
}
if (hdr.ipv6.isValid()) {
if (hdr.ipv6.hop_limit <= 8w1) {
egress.drop = true;
return;
}
}
} else {
attached.apply(ingress, hdr);
}
if (hdr.geneve.isValid() == false) {
nat.apply(hdr, ingress, egress); // check for ingress nat
}
}

if (is_mcast) {
if (hdr.vlan.isValid()) {
hdr.ethernet.ether_type = hdr.vlan.ether_type;
hdr.vlan.setInvalid();
//
// After local and NAT processing, basic packet forwarding happens.
//
// Unicast and multicast are bifurcated, mirroring dendrite's
// sidecar.p4 (the meta.is_mcast metadata check). Multicast traffic
// is replicated and bypasses the unicast nexthop resolver,
// whose default_action punts to the scrimlet. Each replica's
// destination and source MACs are derived in the egress control
// instead.
//
// IPv6 link-local multicast (ff02::/16) never reaches this point; it is
// marked local and handled by the earlier local path. The parser
// rejects interface-local scope multicast, while expired multicast
// (TTL/hop_limit <= 1) gets dropped on the non-local path above.
//
// IPv4 local network control (224.0.0.0/24, RFC 5771 section 4) is
// not dropped by address. The TTL check covers IGMP, OSPF and
// PIM, but senders such as mDNS and VRRP can use TTL 255.
is_mcast = false;
if (hdr.ipv6.isValid()) {
if (hdr.ipv6.dst[127:120] == 8w0xff) { is_mcast = true; }
}
mcast.apply(hdr, ingress, egress);
// Packet copies that keep their encapsulation are forwarded on the
// outer IPv6 header, losing a hop like dendrite's
// MulticastRouter6 (RFC 8200 section 3). Decapsulated copies
// have the inner header decremented before decap_geneve executes.
if (ingress.mcast_decap) {
validate_mcast_external();
if (egress.drop == false) {
if (hdr.inner_ipv4.isValid()) {
ttl_dec.apply(hdr.inner_ipv4);
if (hdr.ipv4.isValid()) {
if (hdr.ipv4.dst[31:28] == 4w0xe) { is_mcast = true; }
}

if (is_mcast) {
if (hdr.vlan.isValid()) {
hdr.ethernet.ether_type = hdr.vlan.ether_type;
hdr.vlan.setInvalid();
}
mcast.apply(hdr, ingress, egress);
// Packet copies that keep their encapsulation are forwarded on the
// outer IPv6 header, losing a hop like dendrite's
// MulticastRouter6 (RFC 8200 section 3). Decapsulated copies
// have the inner header decremented before decap_geneve executes.
if (ingress.mcast_decap) {
validate_mcast_external();
if (egress.drop == false) {
if (hdr.inner_ipv4.isValid()) {
ttl_dec.apply(hdr.inner_ipv4);
}
if (hdr.inner_ipv6.isValid()) {
hdr.inner_ipv6.hop_limit =
hdr.inner_ipv6.hop_limit - 8w1;
}
decap_geneve();
}
if (hdr.inner_ipv6.isValid()) {
hdr.inner_ipv6.hop_limit =
hdr.inner_ipv6.hop_limit - 8w1;
} else {
if (hdr.ipv6.isValid()) {
if (hdr.ipv6.hop_limit <= 8w1) { egress.drop = true; }
if (egress.drop == false) {
hdr.ipv6.hop_limit = hdr.ipv6.hop_limit - 8w1;
}
}
decap_geneve();
}
// An empty bitmap is equivalent to having no multicast group
// configured, and the packet is dropped.
//
// This matches the default action of dendrite's
// mcast_replication_ipv6 (drop_mcastv6_admin_scoped_no_group).
// Neither pipeline sends an ICMP error for multicast (RFC 1812
// section 4.3.2.7).
if (egress.port_bitmap == 128w0) {
egress.drop = true;
}
} else {
if (hdr.ipv6.isValid()) {
if (hdr.ipv6.hop_limit <= 8w1) { egress.drop = true; }
if (egress.drop == false) {
hdr.ipv6.hop_limit = hdr.ipv6.hop_limit - 8w1;
}
router.apply(hdr, ingress, egress);
// Prevent reflection. Multicast ingress-port suppression is
// handled in the replication bitmap, so this applies to
// unicast only.
if (ingress.port == egress.port) { egress.drop = true; } else {
resolver.apply(hdr, ingress, egress);
}
}
// An empty bitmap is equivalent to having no multicast group
// configured, and the packet is dropped.
//
// This matches the default action of dendrite's
// mcast_replication_ipv6 (drop_mcastv6_admin_scoped_no_group).
// Neither pipeline sends an ICMP error for multicast (RFC 1812
// section 4.3.2.7).
if (egress.port_bitmap == 128w0) {
egress.drop = true;
}

//
// Unicast has its source MAC rewritten for that port, but
// multicast is disregarded, as replicas get their MACs in the
// egress control, and multicast packets from the scrimlet
// continue on as userspace had sent it.
if (egress.drop == false) {
if (is_mcast == false) {
// Packets sent to scrimlet don't need their SMAC rewritten.
// If a resolver miss punts the packet and it gets retransmitted
// by tfportd (upon successful ARP/ND resolution), the SMAC is
// rewritten after fwd_from_scrimlet().
if (egress.port != 16w0) { mac.apply(hdr, egress); }
Comment thread
zeeshanlakhani marked this conversation as resolved.
Resolved
}
} else {
router.apply(hdr, ingress, egress);
resolver.apply(hdr, egress);
mac.apply(hdr, egress);

// Prevent reflection. Multicast ingress-port suppression is
// handled in the replication bitmap, so this applies to
// unicast only.
if (ingress.port == egress.port) { egress.drop = true; }
}
}

Expand Down Expand Up @@ -539,26 +557,51 @@ control attached(

control resolver(
inout headers_t hdr,
in ingress_metadata_t ingress,
inout egress_metadata_t egress,
) {
table resolver_v4 {
key = { egress.nexthop_v4: exact; }
actions = { rewrite_dst; drop; }
default_action = drop;
actions = { request_arp; rewrite_dst; drop; }
default_action = request_arp;
}

table resolver_v6 {
key = { egress.nexthop_v6: exact; }
actions = { rewrite_dst; drop; }
default_action = drop;
actions = { request_ndp; rewrite_dst; drop; }
default_action = request_ndp;
}

apply {
if (egress.nexthop_v4 != 32w0) { resolver_v4.apply(); }
if (egress.nexthop_v6 != 128w0) { resolver_v6.apply(); }
}

action request_arp() {
hdr.sidecar.setValid();
hdr.sidecar.sc_ether_type = hdr.ethernet.ether_type;
hdr.ethernet.ether_type = 16w0x0901; // ETHERTYPE_SIDECAR
hdr.sidecar.sc_code = 8w0x03; // SC_ARP_NEEDED
hdr.sidecar.sc_ingress = ingress.port;
hdr.sidecar.sc_egress = egress.port;
hdr.sidecar.sc_payload = 128w0;
hdr.sidecar.sc_payload[31:0] = egress.nexthop_v4;
egress.port = 16w0; // USER_SPACE_SERVICE_PORT
}

action request_ndp() {
hdr.sidecar.setValid();
hdr.sidecar.sc_ether_type = hdr.ethernet.ether_type;
hdr.ethernet.ether_type = 16w0x0901; // ETHERTYPE_SIDECAR
hdr.sidecar.sc_code = 8w0x04; // SC_NEIGHBOR_NEEDED
hdr.sidecar.sc_ingress = ingress.port;
hdr.sidecar.sc_egress = egress.port;
hdr.sidecar.sc_payload = egress.nexthop_v6;
egress.port = 16w0; // USER_SPACE_SERVICE_PORT
}

action rewrite_dst(bit<48> dst) { hdr.ethernet.dst = dst; }

action drop() { egress.drop = true; }
}

Expand Down
Loading