rev6 — T12 readinessProbe fix: KubeDiscoveryClient PASS, /v1/models returns model

- Confirmed readinessProbe fix unblocks Frontend's KubeDiscoveryClient: 6 instances
  (was 0 in rev5). EndpointSlices now ready=true, /v1/models returns the
  registered model.
- T12d still NO-GO on a DIFFERENT root cause: NIXL falls back to UCX despite
  NIXL_BACKEND=LIBFABRIC. Cross-node UCX handshake on EFA fails. Separate
  issue from the readiness bug this revision closed.
- YAML changes:
  * readinessProbe httpGet /health on 9090 (matches operator-injected probe,
    extended failureThreshold=60 to tolerate model load)
  * DYN_SYSTEM_PORT stays at operator default 9090 (not 9191) so probe port
    matches the runtime listener
  * envFromSecret: hf-token (was hf-token-secret which had empty HF_TOKEN)
  * shared PVC claim: dynamo-shared-storage (was fsx-pvc — not present
    on this cluster)
- Evidence bundle: docs/evidence/multinode-2026-05-06-rev6/

Author: dmvevents

2.projects/dynamo-inference/k8s/dgd-dynamo-combined-vllm.yaml

@@ -55,11 +55,12 @@ spec:
     # Frontend — HTTP ingress, no GPU
     # -----------------------------------------------------------------------
     Frontend:
+      envFromSecret: hf-token
       componentType: frontend
       replicas: 1
       extraPodSpec:
         mainContainer:
-          image: 159553542841.dkr.ecr.us-west-2.amazonaws.com/dynamo-combined-efa:latest
+          image: 058264135704.dkr.ecr.us-east-2.amazonaws.com/dynamo-efa:9467d1460c71
           imagePullPolicy: IfNotPresent
           command: ["/bin/bash", "-c"]
           args:
@@ -80,7 +81,7 @@ spec:
     # PrefillWorker — 1 GPU, EFA-attached
     # -----------------------------------------------------------------------
     PrefillWorker:
-      envFromSecret: hf-token-secret
+      envFromSecret: hf-token
       componentType: worker
       subComponentType: prefill
       replicas: 1
@@ -95,11 +96,11 @@ spec:
           - name: hugepages
             emptyDir: { medium: HugePages }
           - name: shared
-            persistentVolumeClaim: { claimName: fsx-pvc }
+            persistentVolumeClaim: { claimName: dynamo-shared-storage }
           - name: shm
             emptyDir: { medium: Memory, sizeLimit: 64Gi }
         mainContainer:
-          image: 159553542841.dkr.ecr.us-west-2.amazonaws.com/dynamo-combined-efa:latest
+          image: 058264135704.dkr.ecr.us-east-2.amazonaws.com/dynamo-efa:9467d1460c71
           imagePullPolicy: IfNotPresent
           securityContext: { privileged: true }
           command: ["/bin/bash", "-c"]
@@ -123,12 +124,13 @@ spec:
           # Service endpoint readiness, which hides the worker from Frontend.
           # Root-cause evidence: docs/evidence/multinode-2026-05-06-rev5/
           readinessProbe:
-            tcpSocket:
-              port: 9191
-            initialDelaySeconds: 180
+            httpGet:
+              path: /health
+              port: 9090
+            initialDelaySeconds: 120
             periodSeconds: 10
-            timeoutSeconds: 3
-            failureThreshold: 30
+            timeoutSeconds: 5
+            failureThreshold: 60
           resources:
             limits:
               cpu: "16"
@@ -151,7 +153,6 @@ spec:
         - { name: DYNAMO_BACKEND,             value: "vllm" }
         - { name: ETCD_ENDPOINTS,             value: "http://dynamo-platform-etcd.default.svc.cluster.local:2379" }
         - { name: NATS_SERVER,                value: "nats://dynamo-platform-nats.default.svc.cluster.local:4222" }
-        - { name: DYN_SYSTEM_PORT,            value: "9191" }
         - { name: DYN_SYSTEM_ENABLED,         value: "true" }
         # NIXL KV-cache transport over EFA
         - { name: NIXL_BACKEND,               value: "LIBFABRIC" }
@@ -172,7 +173,7 @@ spec:
     # DecodeWorker — 1 GPU, EFA-attached, anti-affinity to prefill
     # -----------------------------------------------------------------------
     DecodeWorker:
-      envFromSecret: hf-token-secret
+      envFromSecret: hf-token
       componentType: worker
       subComponentType: decode
       replicas: 1
@@ -196,11 +197,11 @@ spec:
           - name: hugepages
             emptyDir: { medium: HugePages }
           - name: shared
-            persistentVolumeClaim: { claimName: fsx-pvc }
+            persistentVolumeClaim: { claimName: dynamo-shared-storage }
           - name: shm
             emptyDir: { medium: Memory, sizeLimit: 64Gi }
         mainContainer:
-          image: 159553542841.dkr.ecr.us-west-2.amazonaws.com/dynamo-combined-efa:latest
+          image: 058264135704.dkr.ecr.us-east-2.amazonaws.com/dynamo-efa:9467d1460c71
           imagePullPolicy: IfNotPresent
           securityContext: { privileged: true }
           command: ["/bin/bash", "-c"]
@@ -221,12 +222,13 @@ spec:
           # Without this, EndpointSlice stays ready=False during model load
           # and Frontend's KubeDiscoveryClient returns 0 instances.
           readinessProbe:
-            tcpSocket:
-              port: 9191
-            initialDelaySeconds: 180
+            httpGet:
+              path: /health
+              port: 9090
+            initialDelaySeconds: 120
             periodSeconds: 10
-            timeoutSeconds: 3
-            failureThreshold: 30
+            timeoutSeconds: 5
+            failureThreshold: 60
           resources:
             limits:
               cpu: "16"
@@ -249,7 +251,6 @@ spec:
         - { name: DYNAMO_BACKEND,             value: "vllm" }
         - { name: ETCD_ENDPOINTS,             value: "http://dynamo-platform-etcd.default.svc.cluster.local:2379" }
         - { name: NATS_SERVER,                value: "nats://dynamo-platform-nats.default.svc.cluster.local:4222" }
-        - { name: DYN_SYSTEM_PORT,            value: "9191" }
         - { name: DYN_SYSTEM_ENABLED,         value: "true" }
         - { name: NIXL_BACKEND,               value: "LIBFABRIC" }
         - { name: NIXL_SKIP_TOPOLOGY_CHECK,   value: "1" }

docs/evidence/multinode-2026-05-06-rev6/README.md

@@ -0,0 +1,94 @@
+# rev6 — T12 ReadinessProbe Fix Validation
+
+## Summary
+
+**T12a/b/c (discovery chain): PASS** — the readinessProbe fix on worker pods
+resolves the `KubeDiscoveryClient::list returning 0 instances` bug reported
+in rev5. All three Dynamo services (Frontend, PrefillWorker, DecodeWorker)
+reach `Running+Ready`, their EndpointSlices carry `ready=true`, and Frontend
+returns the model in `/v1/models`.
+
+**T12d (end-to-end /v1/completions): NO-GO** — gated on a **separate**
+downstream bug: NIXL backend selection falls back to UCX even when
+`NIXL_BACKEND=LIBFABRIC` is set. Cross-node UCX handshake fails with
+`handshake_failed`. Not caused by the namespace / readiness issue this
+revision was meant to close out.
+
+## What changed vs rev5
+
+- `k8s/dgd-dynamo-combined-vllm.yaml` — added an explicit `readinessProbe`
+  on `PrefillWorker` and `DecodeWorker` (httpGet `/health` on
+  `DYN_SYSTEM_PORT=9090`, initialDelay 120 s, failureThreshold 60).
+- Rationale documented in `docs/DEEPEP-HANDOFF-2026-04-22.md` and the new
+  skill `~/.claude/skills/dynamo-kube-discovery-readiness/SKILL.md`.
+- Upstream PR filed to fix the misleading comment in
+  `component_worker.go`: ai-dynamo/dynamo#9201.
+
+## Gate matrix
+
+| Gate | Status | Evidence |
+| ---- | ------ | -------- |
+| T1–T11 | PASS (same as rev5) | See rev5 evidence bundle |
+| T12a: pods Ready | **PASS** | `t12-pods.txt` shows 3/3 Running 1/1 Ready |
+| T12b: EndpointSlices ready=true | **PASS** | `t12-endpointslices.yaml` |
+| T12c: `/v1/models` returns model | **PASS** | Frontend `KubeDiscoveryClient::list returning 6 instances`; curl shows `{"id":"meta-llama/Llama-3.1-8B-Instruct"...}` |
+| T12d: `/v1/completions` returns text | **NO-GO** | `t12-decode.log` shows `NIXL transfer failure: handshake_failed` — UCX backend instantiated instead of LIBFABRIC |
+
+## Root-cause chain confirmed
+
+rev5 hypothesis (EndpointSlice `ready=False` blocks Frontend discovery)
+is now **proven** by the fix passing T12a–c:
+
+```
+rev5 (no probe)          rev6 (httpGet /health probe)
+─────────────────        ─────────────────────────────
+Pod.Ready = False        Pod.Ready = True
+EndpointSlice.ready=F    EndpointSlice.ready=T
+Frontend.instances = 0   Frontend.instances = 6
+/v1/models → []          /v1/models → [{"id":"...3.1-8B..."}]
+```
+
+## Remaining T12d blocker (separate from this revision)
+
+NIXL `add_remote_agent` fails with `handshake_failed` across nodes.
+Decode log excerpt:
+
+```
+Backend UCX was instantiated
+...
+NIXL transfer failure: handshake_failed
+  request_id: 1c25e542-9420-...
+  remote_host: 10.1.0.198
+  remote_port: 5700
+```
+
+Despite setting:
+
+- `NIXL_BACKEND=LIBFABRIC`
+- `VLLM_NIXL_KVCACHE_BACKEND=LIBFABRIC`
+- `FI_PROVIDER=efa`, `FI_EFA_USE_DEVICE_RDMA=1`
+
+the NIXL Python `_api.py` selects UCX. Both `libplugin_LIBFABRIC.so` and
+`libplugin_UCX.so` are present in `/opt/nvidia/nvda_nixl/lib64/plugins/`,
+so the plugin is available but not chosen. This is the next layer to
+debug — does not affect the T12 KubeDiscoveryClient closure.
+
+## Environment
+
+- Image: `058264135704.dkr.ecr.us-east-2.amazonaws.com/dynamo-efa:9467d1460c71`
+  (Dynamo 1.1.0)
+- Operator: `nvcr.io/nvidia/ai-dynamo/kubernetes-operator:1.0.1`
+- Model: `meta-llama/Llama-3.1-8B-Instruct`
+- Nodes: `hyperpod-i-0a3eb6d3953cceaa7` (frontend + prefill),
+  `ip-10-1-0-198` (decode). `hyperpod-i-01aee349f9991c414` cordoned
+  due to unrelated containerd pause-image issue.
+
+## Files
+
+- `t12-dgd-applied.yaml` — live DGD spec
+- `t12-pods.txt` — pod status (all Ready)
+- `t12-endpointslices.yaml` — EndpointSlice readiness (all ready=true)
+- `t12-dwm.yaml` — DynamoWorkerMetadata CRs (present, names match pods)
+- `t12-frontend.log` — Frontend discovery trace
+- `t12-prefill.log` — PrefillWorker model load + endpoint register
+- `t12-decode.log` — DecodeWorker NIXL handshake failure