From 5c525df3b2c1b646b69915a7d36dc06f075ca314 Mon Sep 17 00:00:00 2001 From: JLCode-tech Date: Fri, 1 May 2026 14:45:48 +1000 Subject: [PATCH] =?UTF-8?q?fix(hp-nodes):=20skip=20userdata=20reboot=20in?= =?UTF-8?q?=20kernel=20mode=20=E2=80=94=20fix=20EKS=20join=20failure?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit In kernel mode (default), the userdata's `nohup bash -c 'sleep 60; reboot' &` fires concurrently with the EKS bootstrap script. Kubelet starts registering the node, then reboot interrupts mid-registration. The nodegroup ends up ACTIVE in AWS but the nodes never appear in `kubectl get nodes`, ultimately flipping to NodeCreationFailure. The reboot exists to make grub-set kernel cmdline params take effect (hugepagesz, isolcpus, nohz_full, rcu_nocbs). But: - Hugepages are already allocated at runtime via sysfs writes immediately below this block, and persisted across future operator-triggered reboots by the systemd-enabled dpdk-hugepages.service. - isolcpus / nohz_full / rcu_nocbs are TMM jitter optimizations — TMM still works correctly without them via cpuset pinning in the pod spec. In sriov mode the reboot path is preserved unchanged: vfio-pci binding genuinely requires a fresh kernel cmdline, and the dpdk-continuation service coordinates a kubelet restart post-reboot. --- .../scripts/compact_userdata.sh | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/infra/aws/high-performance-nodes/scripts/compact_userdata.sh b/infra/aws/high-performance-nodes/scripts/compact_userdata.sh index 01ba95c..a767938 100644 --- a/infra/aws/high-performance-nodes/scripts/compact_userdata.sh +++ b/infra/aws/high-performance-nodes/scripts/compact_userdata.sh @@ -57,13 +57,22 @@ if ! is_checkpoint_complete "kernel_params"; then fi if grep -q "hugepagesz=2M" /proc/cmdline; then NEEDS_REBOOT=false - else + elif [ "$TMM_DATA_PLANE_MODE" = "sriov" ]; then NEEDS_REBOOT=true mkdir -p /etc/default/grub.d echo "GRUB_CMDLINE_LINUX=\"$${GRUB_CMDLINE_LINUX:-} $KP\"" > /etc/default/grub.d/99-dpdk-hugepages.cfg cp /etc/default/grub /etc/default/grub.backup grep -q "hugepagesz=2M" /etc/default/grub || sed -i "s/biosdevname=0/& $KP/g" /etc/default/grub grub2-mkconfig -o /boot/grub2/grub.cfg + else + # Kernel mode: skip grub mutation. The 60s-delayed reboot below races with the EKS + # bootstrap script and interrupts kubelet's first registration. Hugepages are + # allocated at runtime via sysfs (below) and persisted across future operator-triggered + # reboots by the systemd-enabled dpdk-hugepages.service. isolcpus/nohz_full/rcu_nocbs + # would require a reboot to take effect but they are TMM jitter optimizations only — + # TMM still works via cpuset pinning in the pod spec. + NEEDS_REBOOT=false + log "kernel mode: skipping grub mutation, no reboot will be scheduled" fi echo $HUGEPAGES_2MI > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages 2>/dev/null || true echo $HUGEPAGES_1GI > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages 2>/dev/null || true