Skip to content

test(ci): summarize nested kubelet failures with a closed set of signals - #8303

Merged
devantler merged 2 commits into
mainfrom
claude/ci-nested-kubelet-signals-8186
Nov 11, 2026
Merged

devantler merged 2 commits into
mainfrom
claude/ci-nested-kubelet-signals-8186

Conversation

@devantler

Copy link
Copy Markdown
Contributor

Why

When a nested Kind cluster fails to start in the Kubernetes-provider trial, the saved evidence shows that the nested kubelet failed but not why. An earlier attempt kept giving misleading labels, because it searched the whole journal for loose phrases.

What

Failed trials now print a short summary of the nested kubelet journal, using only labels from a fixed list. A label is given only when all of its phrases appear on the same journal line, and no journal text is ever published.

Fixes #8186

🤖 Generated with Claude Code

When a nested Kind cluster fails in the Kubernetes-provider trial, the
retained evidence showed that the nested kubelet failed but not why.

Add a closed classifier for the nested kubelet journal: it keeps the
newest bounded bytes and drops a line cut part-way, matches each
signal only when all its phrases are on the same line, emits only
labels from one allow-list, and reports empty input as unavailable.
The trial prints it for the control-plane and worker nodes of a
failed Vanilla cluster. A test checks every output line against the
allow-list and that no journal text is published.

Part of #8186

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

✅⚠️MegaLinter analysis: Success with warnings

⚠️ BASH / bash-exec - 1 error
Results of bash-exec linter (version 5.3.9)
See documentation on https://megalinter.io/10.2.0/descriptors/bash_bash_exec/
-----------------------------------------------

✅ [SUCCESS] .github/actions/cleanup-hetzner/cleanup-hetzner.sh
✅ [SUCCESS] .github/actions/cleanup-hetzner/cleanup-hetzner.test.sh
✅ [SUCCESS] .github/actions/free-disk-space/free-disk-space.sh
✅ [SUCCESS] .github/actions/ksail-cluster/hetzner-inventory-config.sh
✅ [SUCCESS] .github/actions/ksail-system-test/argocd-adoption.sh
✅ [SUCCESS] .github/actions/ksail-system-test/autoscaler-audit-trial.sh
✅ [SUCCESS] .github/actions/ksail-system-test/calico-migration.sh
✅ [SUCCESS] .github/actions/ksail-system-test/calico-migration.test.sh
✅ [SUCCESS] .github/actions/ksail-system-test/chart-values-drift.sh
✅ [SUCCESS] .github/actions/ksail-system-test/helm-cli.sh
✅ [SUCCESS] .github/actions/ksail-system-test/helm-values-precedence.sh
✅ [SUCCESS] .github/actions/ksail-system-test/offline-ecr-upstream.sh
✅ [SUCCESS] .github/actions/ksail-system-test/offline-ecr-upstream.test.sh
✅ [SUCCESS] .github/actions/ksail-system-test/talos-kubernetes-upgrade.sh
✅ [SUCCESS] .github/actions/ksail-system-test/verify-offline-ecr-pull.sh
✅ [SUCCESS] .github/actions/ksail-system-test/verify-offline-ecr-pull.test.sh
✅ [SUCCESS] .github/actions/ksail-test-workload/install-with-recovery.sh
✅ [SUCCESS] .github/actions/ksail-test-workload/install-with-recovery.test.sh
✅ [SUCCESS] .github/actions/ksail-test-workload/reject-remote-kustomize-resources.sh
✅ [SUCCESS] .github/actions/ksail-test-workload/remote-resource-guard.test.sh
✅ [SUCCESS] .github/actions/pull-registry-image/pull-registry-image.sh
✅ [SUCCESS] .github/actions/rate-limit-gate/wait-for-rate-limit.sh
✅ [SUCCESS] .github/actions/rate-limit-gate/wait-for-rate-limit.test.sh
✅ [SUCCESS] .github/actions/warm-mirror-cache/cache-validation.test.sh
✅ [SUCCESS] .github/actions/warm-mirror-cache/mirror-artifact.sh
✅ [SUCCESS] .github/actions/warm-mirror-cache/pull-helper-image.sh
✅ [SUCCESS] .github/actions/warm-mirror-cache/pull-helper-image.test.sh
✅ [SUCCESS] .github/scripts/assert-gitops-deployed.sh
✅ [SUCCESS] .github/scripts/assert-gitops-deployed.test.sh
✅ [SUCCESS] .github/scripts/assert-k3k-server-creator.sh
✅ [SUCCESS] .github/scripts/assert-k3k-server-creator.test.sh
✅ [SUCCESS] .github/scripts/build-calico-migration-baseline.sh
✅ [SUCCESS] .github/scripts/cask-pr-handoff-workflow.test.sh
✅ [SUCCESS] .github/scripts/cask-read-api.sh
✅ [SUCCESS] .github/scripts/cask-read-api.test.sh
✅ [SUCCESS] .github/scripts/cask-rendering.test.sh
✅ [SUCCESS] .github/scripts/check-eks-capacity.sh
✅ [SUCCESS] .github/scripts/check-eks-capacity.test.sh
✅ [SUCCESS] .github/scripts/codeql-autobuild-prepare.sh
✅ [SUCCESS] .github/scripts/codeql-autobuild-prepare.test.sh
✅ [SUCCESS] .github/scripts/codeql-source.sh
✅ [SUCCESS] .github/scripts/collect-cask-pr-handoff.sh
✅ [SUCCESS] .github/scripts/collect-cask-pr-handoff.test.sh
✅ [SUCCESS] .github/scripts/collect-release-build-experiment.sh
✅ [SUCCESS] .github/scripts/delete-eks-smoke-cluster.sh
✅ [SUCCESS] .github/scripts/delete-eks-smoke-cluster.test.sh
✅ [SUCCESS] .github/scripts/delete-old-workflow-runs.sh
✅ [SUCCESS] .github/scripts/delete-old-workflow-runs.test.sh
✅ [SUCCESS] .github/scripts/diagnostics-artifact-fixture.sh
✅ [SUCCESS] .github/scripts/diagnostics-artifact-fixture.test.sh
✅ [SUCCESS] .github/scripts/diagnostics-artifact-ksail-fixture.sh
✅ [SUCCESS] .github/scripts/download-go-modules.sh
✅ [SUCCESS] .github/scripts/eks-report-test-lib.sh
✅ [SUCCESS] .github/scripts/find-merged-cask-pr-handoff.sh
✅ [SUCCESS] .github/scripts/find-merged-cask-pr-handoff.test.sh
✅ [SUCCESS] .github/scripts/finish-cask-pr-handoff.sh
✅ [SUCCESS] .github/scripts/finish-cask-pr-handoff.test.sh
❌ [ERROR] .github/scripts/generate-completions.sh
    Error: File:[.github/scripts/generate-completions.sh] is not executable

✅ [SUCCESS] .github/scripts/go-notices-release.test.sh
✅ [SUCCESS] .github/scripts/install-release.test.sh
✅ [SUCCESS] .github/scripts/list-link-check-files.sh
✅ [SUCCESS] .github/scripts/prepare-nested-kind-host.sh
✅ [SUCCESS] .github/scripts/prepare-nested-kind-host.test.sh
✅ [SUCCESS] .github/scripts/profile-codeql-cohort.sh
✅ [SUCCESS] .github/scripts/profile-codeql-compare.sh
✅ [SUCCESS] .github/scripts/profile-codeql-compare.test.sh
✅ [SUCCESS] .github/scripts/profile-codeql-files.sh
✅ [SUCCESS] .github/scripts/profile-codeql-files.test.sh
✅ [SUCCESS] .github/scripts/profile-codeql-go.sh
✅ [SUCCESS] .github/scripts/profile-codeql-go.test.sh
✅ [SUCCESS] .github/scripts/profile-codeql-swap.sh
✅ [SUCCESS] .github/scripts/profile-codeql-swap.test.sh
✅ [SUCCESS] .github/scripts/publish-mcp-registry.sh
✅ [SUCCESS] .github/scripts/publish-mcp-registry.test.sh
✅ [SUCCESS] .github/scripts/publish-operator-chart.sh
✅ [SUCCESS] .github/scripts/redraft-evergreen-cask-prs.sh
✅ [SUCCESS] .github/scripts/redraft-evergreen-cask-prs.test.sh
✅ [SUCCESS] .github/scripts/release-build-collector.test.sh
✅ [SUCCESS] .github/scripts/release-build-experiment.test.sh
✅ [SUCCESS] .github/scripts/release-build-samples.lib.sh
✅ [SUCCESS] .github/scripts/report-eks-account-capacity.sh
✅ [SUCCESS] .github/scripts/report-eks-account-capacity.test.sh
✅ [SUCCESS] .github/scripts/report-eks-create-failure.sh
✅ [SUCCESS] .github/scripts/report-eks-create-failure.test.sh
✅ [SUCCESS] .github/scripts/resolve-cask-publication.sh
✅ [SUCCESS] .github/scripts/resolve-cask-publication.test.sh
✅ [SUCCESS] .github/scripts/resolve-release-publish-state.sh
✅ [SUCCESS] .github/scripts/resolve-release-publish-state.test.sh
✅ [SUCCESS] .github/scripts/resolve-release-replay.sh
✅ [SUCCESS] .github/scripts/strict-link-transport.sh
✅ [SUCCESS] .github/scripts/strict-link-transport.test.sh
✅ [SUCCESS] .github/scripts/style-clean-cask-branch.sh
✅ [SUCCESS] .github/scripts/style-clean-cask-branch.test.sh
✅ [SUCCESS] .github/scripts/summarize-mutation-report.sh
✅ [SUCCESS] .github/scripts/summarize-mutation-report.test.sh
✅ [SUCCESS] .github/scripts/summarize-nested-kubelet.sh
✅ [SUCCESS] .github/scripts/upload-release-assets.sh
✅ [SUCCESS] .github/scripts/upload-release-assets.test.sh
✅ [SUCCESS] .github/scripts/validate-cask-pr-handoff.sh
✅ [SUCCESS] .github/scripts/validate-cask-pr-handoff.test.sh
✅ [SUCCESS] .github/scripts/validate-release-ref.sh
✅ [SUCCESS] .github/scripts/validate-release-ref.test.sh
✅ [SUCCESS] .github/scripts/verify-desktop-codeql.sh
✅ [SUCCESS] .github/scripts/verify-desktop-codeql.test.sh
✅ [SUCCESS] .github/scripts/verify-diagnostics-artifact.sh
✅ [SUCCESS] .github/scripts/verify-go-archive-parity.sh
✅ [SUCCESS] .github/scripts/verify-go-archive-parity.test.sh
✅ [SUCCESS] .github/scripts/verify-otelzap-parity.sh
✅ [SUCCESS] .github/scripts/verify-otelzap-parity.test.sh
✅ [SUCCESS] .github/scripts/verify-vendored-comment-scan.sh
✅ [SUCCESS] .github/scripts/wait-for-github-api-budget.sh
✅ [SUCCESS] .github/scripts/wait-for-github-api-budget.test.sh
✅ [SUCCESS] desktop/scripts/make-icns.sh
✅ [SUCCESS] desktop/scripts/make-info-plist.sh
✅ [SUCCESS] desktop/scripts/make-macos-app.sh
✅ [SUCCESS] docs/demos/render.sh
✅ [SUCCESS] images/ksail-analysis-runner/smoke.sh
✅ [SUCCESS] install.sh
✅ [SUCCESS] internal/ciharness/testdata/schematic_fake_ksail.sh
✅ [SUCCESS] pkg/svc/provisioner/cluster/vcluster/containerd_registry_hosts.sh
✅ [SUCCESS] scripts/stage-webui.sh

✅ Linters with no issues

actionlint, git_diff, hadolint, jscpd, jsonlint, lychee, markdown-table-formatter, markdownlint, prettier, prettier, shellcheck, shfmt, stylelint, syft, trivy-sbom, trufflehog, v8r, v8r, yamllint

See detailed reports in MegaLinter artifacts

MegaLinter is provided by OX Security
Show us your support by starring ⭐ the repository

@github-code-quality

github-code-quality Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Code coverage overview

Languages: Go

Go / code-coverage/go

main a0c9893 claude/ci-nested-kub... 55a5b75 +/-
72% 72% 0%
View coverage of the 1 most impacted file.
File main a0c9893 claude/ci-nested-kub... 55a5b75 +/-
pkg/cli/cluster...i/distconfig.go 80% 78% -2%

ℹ️ Calculations represent line coverage. Results may change as calculations are updated.


Updated November 10, 2026 22:59 UTC

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer (interactive Claude Code session)

User trial at 930125268e5278a595466743f19367e7bf2a0a82

  • Ran the summary script by hand the way the failed-trial step feeds it. A journal with an fsnotify "too many open files" line and a "validate service connection" line printed exactly two fixed labels (inotify-instance-limit, runtime-connection) and none of the journal text. Empty input printed unavailable; free text with no known signal printed unclassified.
  • The classifier test passes locally (16 cases): every output line must be an allowed label, phrases on different lines never combine, an inotify permission error is not a watch limit, the newest bytes are kept and a line cut part-way is dropped.
  • The new step only runs when a nested Vanilla trial has already failed, so this head's green CI does not execute it live. If the journal read fails, the step prints that evidence is unavailable and the trial outcome is unchanged. The optional overlapping/disjoint network report from the issue is not part of this change.

🤖 Generated with Claude Code

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-11T00:08:03.175797Z 55a5b75 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 930125268e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

> 🤖 Generated by the Agentic Engineer

Match the explicit unknown-driver error emitted by kubelet. Cover successful CRI driver discovery alone and beside an unrelated failure, plus the actual driver error.
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer (interactive Claude Code session)

User trial at 55a5b755faa08b858775ea3c88d1d6d8a1d0b0b3

  • The Codex finding is fixed at this head (55a5b755f), which narrows the cgroup-driver signal to the kubelet's own error. My earlier reply named a local commit with the same fix; it was not pushed, because this commit already covered it.
  • Ran the summary script by hand at this head: the healthy kubelet line Using cgroup driver setting received from the CRI runtime alone prints unclassified; adding a runtime returned an unknown cgroup driver error prints cgroup-driver. No journal text is printed. The classifier tests pass locally (20 cases) and CI is green (99 checks).
  • The new step still runs only after a nested Vanilla trial has failed, so green CI does not execute it live.

🤖 Generated with Claude Code

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: 55a5b755fa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@devantler
devantler marked this pull request as ready for review November 11, 2026 00:09
@devantler
devantler merged commit cc1b9ac into main Nov 11, 2026
99 checks passed
@devantler
devantler deleted the claude/ci-nested-kubelet-signals-8186 branch November 11, 2026 00:10
@github-project-automation github-project-automation Bot moved this from 🫴 Ready to ✅ Done in 🌊 Project Board Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

Summarize nested kubelet failures with a closed set of fixed signals

1 participant