Skip to content

feat(nvca): add worker-based health per-instance feature#338

Open
estroz wants to merge 5 commits into
mainfrom
estroczynski/feat/worker-health
Open

feat(nvca): add worker-based health per-instance feature#338
estroz wants to merge 5 commits into
mainfrom
estroczynski/feat/worker-health

Conversation

@estroz

@estroz estroz commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

TL;DR

Additional Details (optional for docs, build, test, refactor, ci, chore, style, and revert PRs)

For the Reviewer

For QA (optional for docs, build, test, refactor, ci, chore, style, and revert PRs)

Issues

Checklist

  • I am familiar with the Contributing Guidelines.
  • I have signed off my commits for Developer Certificate of Origin (DCO) compliance.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

Summary by CodeRabbit

  • New Features
    • Added workload-scoped feature flag configuration via the nvcf-workload-config ConfigMap (from config.yaml).
    • Added an option to evaluate MiniService health based on worker readiness.
    • Improved status reporting with more specific degraded-workload vs degraded-worker reasons and a WorkersHealthy condition.
    • Expanded pod diagnostics to surface multiple image-pull issues with container-level details.
  • Bug Fixes
    • Improved image-pull metrics/logs and failure handling for multiple affected containers, including correct timeouts and metrics accounting.

@estroz
estroz requested a review from a team as a code owner July 22, 2026 00:07
@coderabbitai

coderabbitai Bot commented Jul 22, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Changes

MiniService workload configuration and status

Layer / File(s) Summary
Workload configuration contracts and decoder
src/compute-plane-services/nvca/pkg/apis/nvca/v1alpha1/..., src/compute-plane-services/nvca/pkg/featureflag/...
Adds WorkloadConfig, supported workload feature flags, YAML decoding with unknown-key filtering, tests, Bazel wiring, and OpenAPI schema output.
Rendered configuration persistence and controller identity
src/compute-plane-services/nvca/internal/miniservice/controller.go, reconcile.go, revision.go, reconcile_test.go
Extracts workload configuration from rendered ConfigMaps, persists it with server-side apply during installation, updates decode call sites, and centralizes controller naming.
Structured pod diagnostics
src/compute-plane-services/nvca/internal/util/k8sutil/pod.go, pod_test.go
Changes image-pull, degraded, and stuck-initialization helpers to return structured container details.
Worker-readiness status evaluation
src/compute-plane-services/nvca/internal/miniservice/status.go, status_test.go, status_karta_test.go
Adds per-object status collection, worker-container readiness evaluation, structured diagnostic propagation, feature-flag dispatch, and updated condition filtering.
Backend consumers and metrics
src/compute-plane-services/nvca/pkg/nvca/..., src/compute-plane-services/nvca/pkg/storage/modelcache.go
Updates logs, instance states, purge gating, progress detection, and image-pull metrics for structured multi-issue results and separated degraded reasons.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ConfigMap
  participant Reconciler
  participant MiniService
  participant doStatus
  participant UtilsPod
  ConfigMap->>Reconciler: provide workload configuration
  Reconciler->>MiniService: apply spec.workloadConfig
  doStatus->>UtilsPod: inspect worker container readiness
  UtilsPod-->>doStatus: return readiness and pod diagnostics
  doStatus->>MiniService: update ObjectsHealthy and WorkersHealthy
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 5.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title uses valid Conventional Commits format and accurately describes the feature-focused change set.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch estroczynski/feat/worker-health

Comment @coderabbitai help to get the list of available commands.

@estroz

estroz commented Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

Note to reviewer: need to fix deepcopy gen and add tests

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 8

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
src/compute-plane-services/nvca/internal/util/k8sutil/pod.go (1)

259-281: 🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win

Nil-pointer dereference plus early-return-in-loop in IsPodStuckInitializing. Both the container and init-container branches read LastTerminationState.Terminated.Message unconditionally even though the preceding switch already tolerates a nil Terminated (using Waiting), which panics; and the len(stuckContainers) > 0 check inside the loop returns after the first match so Containers never accumulates more than one entry, which can cause the worker-name match in doStatusByWorkerReadiness to miss a stuck worker container.

  • src/compute-plane-services/nvca/internal/util/k8sutil/pod.go#L259-L281: guard Message with a nil check on Terminated, and move the len(stuckContainers) > 0 return outside the ContainerStatuses loop.
  • src/compute-plane-services/nvca/internal/util/k8sutil/pod.go#L283-L305: apply the same nil guard on Message and move the len(stuckContainers) > 0 return outside the InitContainerStatuses loop.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/compute-plane-services/nvca/internal/util/k8sutil/pod.go` around lines
259 - 281, Update IsPodStuckInitializing in
src/compute-plane-services/nvca/internal/util/k8sutil/pod.go at lines 259-281
and 283-305: in both ContainerStatuses and InitContainerStatuses handling, only
read Terminated.Message when Terminated is non-nil, and move the
len(stuckContainers) > 0 return until after each complete loop so all matching
containers are accumulated before returning.
🧹 Nitpick comments (4)
src/compute-plane-services/nvca/internal/miniservice/reconcile_test.go (1)

4317-4320: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Cover the reserved ConfigMap path.

Add a nvcf-workload-feature-flags fixture that asserts decoded flags are returned and that this ConfigMap is excluded from both objs and resources. The current test discards the new return value and exercises only an ordinary ConfigMap.

As per coding guidelines, “Code changes must include tests.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/compute-plane-services/nvca/internal/miniservice/reconcile_test.go`
around lines 4317 - 4320, Update the decodeObjects test around the existing
ordinary ConfigMap fixture to add a nvcf-workload-feature-flags ConfigMap case,
capture and assert the decoded feature flags, and verify that the reserved
ConfigMap is excluded from both objs and resources. Keep the existing ordinary
ConfigMap assertions intact and ensure the test covers the newly returned value.

Source: Coding guidelines

src/compute-plane-services/nvca/internal/miniservice/reconcile.go (1)

468-469: 📐 Maintainability & Code Quality | 🔵 Trivial

Trace the MiniService feature-flag patch. saveWorkloadFeatureFlags performs a Kubernetes API write without its own service.operation span. Add a child span around this patch and record errors on the span so the call is observable.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/compute-plane-services/nvca/internal/miniservice/reconcile.go` around
lines 468 - 469, Update saveWorkloadFeatureFlags around the r.Client.Patch call
to create a child service.operation span, ensure it is ended, and record any
patch error on the span before returning it. Preserve the existing wrapped error
behavior and use the surrounding tracing conventions.

Sources: Coding guidelines, Path instructions

src/compute-plane-services/nvca/internal/miniservice/status.go (2)

513-517: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Track this follow-up in a ticket rather than an inline TODO.

Per repo guidelines, follow-up work should be filed as a ticket instead of left as a TODO in code. Want me to open one for "worker-readiness: use only utils container readiness and fail on restart count > 3"?

As per coding guidelines: "file a ticket for follow-up work instead of leaving a TODO in code."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/compute-plane-services/nvca/internal/miniservice/status.go` around lines
513 - 517, Remove the inline TODO block near the worker readiness logic in
status.go, and track its two follow-up items—using only utils container
readiness and failing when the utils container restart count exceeds 3—in the
repository’s ticketing system instead.

Source: Coding guidelines


441-606: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy lift

Add tests for the new doStatusByWorkerReadiness path.

This introduces a substantial new feature-flagged status branch (worker readiness, degraded/stuck/image-pull worker detection, pending-timeout), but status_test.go only updates a reason constant. Per guidelines, code changes should include tests or the PR should explain why they're not applicable (the PR is a draft with no such note). I can draft table-driven cases (worker ready, degraded worker, stuck init, image-pull, pending timeout, utils-not-found) if helpful.

As per coding guidelines: "Code changes must include tests, or the Pull Request must explain why tests are not applicable."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/compute-plane-services/nvca/internal/miniservice/status.go` around lines
441 - 606, Add table-driven tests in status_test.go covering
doStatusByWorkerReadiness for a ready worker, degraded worker, stuck
initialization, image-pull issues, pending timeout, and a missing utils pod.
Assert the returned reconcile result/error and updated MiniService phase and
worker-health condition for each case, using existing test helpers and status
constants.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/compute-plane-services/nvca/internal/miniservice/reconcile.go`:
- Around line 1867-1870: Update the error path in the ConfigMap handling around
DecodeWorkloadFeatureFlags so malformed recognized flags are wrapped with
contextual information using reconcile.TerminalError before returning. Preserve
successful decoding and existing return values, ensuring immutable invalid
rendered data stops retrying until the workload changes.

In `@src/compute-plane-services/nvca/internal/util/k8sutil/pod_test.go`:
- Around line 709-712: Update the test assertions around ImagePullIssuesReported
so imagePullIssues length and reason are checked only when tt.wantIssue is true.
Keep the hasIssue equality assertion unconditional, while preserving the
existing length-one and reason checks for expected issues.

In `@src/compute-plane-services/nvca/pkg/apis/nvca/v1alpha1/miniservice_types.go`:
- Around line 57-64: Add Godoc comments for the exported WorkloadFeatureFlag
interface and the BooleanWorkloadFeatureFlag IsEnabled and Get methods, ensuring
each comment begins with its exact symbol name. Correct the existing
WorkloadFeatureFlags.IsEnabled comment so it documents IsEnabled rather than
Get, while preserving its stated behavior.

In
`@src/compute-plane-services/nvca/pkg/featureflag/featureflag_workload_test.go`:
- Around line 74-101: Update the DecodeWorkloadFeatureFlags tests in the
table-driven loop: compare the recognized flag’s enabled value via
IsEnabled(StatusByWorkerReadiness) and assert the decoded map length instead of
comparing the map directly to a bool. For the “mix of valid, unknown, and
invalid” case, remove the expected error and assert successful decoding with
only the recognized flag enabled, ignoring unknown entries such as
ANOTHER_UNKNOWN.

In `@src/compute-plane-services/nvca/pkg/featureflag/featureflag_workload.go`:
- Line 67: Replace the TODO comment about supporting non-boolean feature flag
types with a reference to a tracked follow-up issue, using the project’s
established ticket-link format; do not leave an untracked TODO in the feature
flag implementation.
- Around line 51-70: Update the DecodeWorkloadFeatureFlags documentation to
state that recognized keys with values that fail boolean parsing return an error
rather than being ignored and warned. Rename the corresponding test case
describing invalid boolean values so it expects and clearly reflects the error
behavior, while preserving handling of unknown keys.
- Around line 61-64: Update the unknown-key warning in the workload feature-flag
loop to use the structured logging API on the logger returned by core.GetLogger,
attaching the feature-flag key and WorkloadFeatureFlagsConfigMapName as fields
instead of interpolating them into the message. Preserve the existing warning
level and logger context.

In `@src/compute-plane-services/nvca/pkg/nvca/k8scomputebackend.go`:
- Around line 1349-1360: Update the header construction around sb.WriteString so
ordinary prefixes retain the existing newline before the "---" separator, while
image-pull issue prefixes remain correctly separated without duplicate blank
lines. Add regression tests covering both a plain prefix and a prefix containing
image-pull messages, including the expected serialized output.

---

Outside diff comments:
In `@src/compute-plane-services/nvca/internal/util/k8sutil/pod.go`:
- Around line 259-281: Update IsPodStuckInitializing in
src/compute-plane-services/nvca/internal/util/k8sutil/pod.go at lines 259-281
and 283-305: in both ContainerStatuses and InitContainerStatuses handling, only
read Terminated.Message when Terminated is non-nil, and move the
len(stuckContainers) > 0 return until after each complete loop so all matching
containers are accumulated before returning.

---

Nitpick comments:
In `@src/compute-plane-services/nvca/internal/miniservice/reconcile_test.go`:
- Around line 4317-4320: Update the decodeObjects test around the existing
ordinary ConfigMap fixture to add a nvcf-workload-feature-flags ConfigMap case,
capture and assert the decoded feature flags, and verify that the reserved
ConfigMap is excluded from both objs and resources. Keep the existing ordinary
ConfigMap assertions intact and ensure the test covers the newly returned value.

In `@src/compute-plane-services/nvca/internal/miniservice/reconcile.go`:
- Around line 468-469: Update saveWorkloadFeatureFlags around the r.Client.Patch
call to create a child service.operation span, ensure it is ended, and record
any patch error on the span before returning it. Preserve the existing wrapped
error behavior and use the surrounding tracing conventions.

In `@src/compute-plane-services/nvca/internal/miniservice/status.go`:
- Around line 513-517: Remove the inline TODO block near the worker readiness
logic in status.go, and track its two follow-up items—using only utils container
readiness and failing when the utils container restart count exceeds 3—in the
repository’s ticketing system instead.
- Around line 441-606: Add table-driven tests in status_test.go covering
doStatusByWorkerReadiness for a ready worker, degraded worker, stuck
initialization, image-pull issues, pending timeout, and a missing utils pod.
Assert the returned reconcile result/error and updated MiniService phase and
worker-health condition for each case, using existing test helpers and status
constants.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 49d77828-1114-4f33-800c-18cc429f3191

📥 Commits

Reviewing files that changed from the base of the PR and between 106c310 and b488878.

📒 Files selected for processing (16)
  • src/compute-plane-services/nvca/internal/miniservice/controller.go
  • src/compute-plane-services/nvca/internal/miniservice/reconcile.go
  • src/compute-plane-services/nvca/internal/miniservice/reconcile_test.go
  • src/compute-plane-services/nvca/internal/miniservice/revision.go
  • src/compute-plane-services/nvca/internal/miniservice/status.go
  • src/compute-plane-services/nvca/internal/miniservice/status_test.go
  • src/compute-plane-services/nvca/internal/util/k8sutil/pod.go
  • src/compute-plane-services/nvca/internal/util/k8sutil/pod_test.go
  • src/compute-plane-services/nvca/pkg/apis/nvca/v1alpha1/miniservice_status_types.go
  • src/compute-plane-services/nvca/pkg/apis/nvca/v1alpha1/miniservice_types.go
  • src/compute-plane-services/nvca/pkg/featureflag/BUILD.bazel
  • src/compute-plane-services/nvca/pkg/featureflag/featureflag_workload.go
  • src/compute-plane-services/nvca/pkg/featureflag/featureflag_workload_test.go
  • src/compute-plane-services/nvca/pkg/nvca/k8scomputebackend.go
  • src/compute-plane-services/nvca/pkg/nvca/k8scomputebackend_miniservice.go
  • src/compute-plane-services/nvca/pkg/storage/modelcache.go

Comment thread src/compute-plane-services/nvca/internal/miniservice/reconcile.go Outdated
Comment thread src/compute-plane-services/nvca/internal/util/k8sutil/pod_test.go
Comment thread src/compute-plane-services/nvca/pkg/apis/nvca/v1alpha1/miniservice_types.go Outdated
Comment thread src/compute-plane-services/nvca/pkg/featureflag/featureflag_workload_test.go Outdated
Comment thread src/compute-plane-services/nvca/pkg/featureflag/featureflag_workload.go Outdated
Comment thread src/compute-plane-services/nvca/pkg/featureflag/featureflag_workload.go Outdated
Comment thread src/compute-plane-services/nvca/pkg/featureflag/featureflag_workload.go Outdated
Comment thread src/compute-plane-services/nvca/pkg/nvca/k8scomputebackend.go Outdated

@kristinapathak kristinapathak left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM other than coderabbit comments, error returned in DecodeWorkloadFeatureFlags().

@estroz
estroz force-pushed the estroczynski/feat/worker-health branch 3 times, most recently from 8c29476 to a5c2d1b Compare July 22, 2026 19:24

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

🧹 Nitpick comments (1)
src/compute-plane-services/nvca/internal/miniservice/reconcile_test.go (1)

662-676: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Cover every directly injected collector setting.

The test verifies only log-chunking values on the utils collector. Assert the metric-subset and workload-metric environment variables there too; metadata assertions do not prove direct sidecar injection works.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/compute-plane-services/nvca/internal/miniservice/reconcile_test.go`
around lines 662 - 676, Extend the BYOO collector environment assertions in the
utils pod test around byooCollectorEnv to cover every directly injected
metric-subset and workload-metric environment variable, using the corresponding
nvcaconfig symbols and expected values. Keep the existing log-chunking and
metadata assertions unchanged.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/compute-plane-services/nvca/internal/miniservice/reconcile.go`:
- Line 956: Update the workload reconciliation flow around decodeObjects to
retain the newly decoded/rendered config, carry it through the update path, and
persist it to MiniService.Spec.WorkloadConfig only after the workload update
succeeds. Ensure status evaluation uses the persisted current config, and add a
regression test covering a chart update that changes StatusByWorkerReadiness.
- Around line 474-475: Instrument the Kubernetes patch operation in the
MiniService reconciliation flow with a stable OpenTelemetry span named
miniservice.save_workload_config. Start and end the span around r.Client.Patch,
and record the patch error while setting the span status to error before
returning the existing wrapped error; preserve successful behavior.

In `@src/compute-plane-services/nvca/pkg/apis/nvca/v1alpha1/BUILD.bazel`:
- Line 27: Add the direct dependency "//vendor/sigs.k8s.io/yaml" to the deps of
the pkg/featureflag BUILD.bazel target, matching the existing declaration in the
referenced BUILD file, so featureflag_workload.go's sigs.k8s.io/yaml import
satisfies Bazel strict-deps.

In `@src/compute-plane-services/nvca/pkg/apis/nvca/v1alpha1/miniservice_types.go`:
- Around line 54-60: Regenerate the API artifacts for the updated WorkloadConfig
type: run make codegen-update and make openapigen-update so FeatureFlags
receives generated map-copy support, saveWorkloadConfig’s desired.DeepCopy()
compiles, and the OpenAPI schema includes workloadConfig; commit all generated
outputs.

In
`@src/compute-plane-services/nvca/pkg/featureflag/featureflag_workload_test.go`:
- Around line 37-109: Update TestDecodeWorkloadConfig’s want field to
*v1alpha1.WorkloadConfig, use nil for nil, missing-key, and empty-config cases,
and take addresses of expected configs in successful cases. Compare got directly
with the pointer expectation so the table matches DecodeWorkloadConfig’s return
type.

In `@src/compute-plane-services/nvca/pkg/featureflag/featureflag_workload.go`:
- Around line 69-70: Update the YAML unmarshalling error path in the workload
configuration loader to wrap the original error with the context “decode
workload config,” preserving the underlying error for unwrapping. Keep
successful parsing and the existing return behavior unchanged.

---

Nitpick comments:
In `@src/compute-plane-services/nvca/internal/miniservice/reconcile_test.go`:
- Around line 662-676: Extend the BYOO collector environment assertions in the
utils pod test around byooCollectorEnv to cover every directly injected
metric-subset and workload-metric environment variable, using the corresponding
nvcaconfig symbols and expected values. Keep the existing log-chunking and
metadata assertions unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2121c6c8-4c95-4ab5-9646-01e9fd9fc60c

📥 Commits

Reviewing files that changed from the base of the PR and between b488878 and 8c29476.

⛔ Files ignored due to path filters (1)
  • src/compute-plane-services/nvca/pkg/apis/nvca/v1alpha1/zz_generated.deepcopy.go is excluded by !**/zz_generated.*
📒 Files selected for processing (19)
  • src/compute-plane-services/nvca/internal/miniservice/controller.go
  • src/compute-plane-services/nvca/internal/miniservice/reconcile.go
  • src/compute-plane-services/nvca/internal/miniservice/reconcile_test.go
  • src/compute-plane-services/nvca/internal/miniservice/revision.go
  • src/compute-plane-services/nvca/internal/miniservice/status.go
  • src/compute-plane-services/nvca/internal/miniservice/status_karta_test.go
  • src/compute-plane-services/nvca/internal/miniservice/status_test.go
  • src/compute-plane-services/nvca/internal/util/k8sutil/pod.go
  • src/compute-plane-services/nvca/internal/util/k8sutil/pod_test.go
  • src/compute-plane-services/nvca/pkg/apis/nvca/v1alpha1/BUILD.bazel
  • src/compute-plane-services/nvca/pkg/apis/nvca/v1alpha1/generated.openapi.go
  • src/compute-plane-services/nvca/pkg/apis/nvca/v1alpha1/miniservice_status_types.go
  • src/compute-plane-services/nvca/pkg/apis/nvca/v1alpha1/miniservice_types.go
  • src/compute-plane-services/nvca/pkg/featureflag/BUILD.bazel
  • src/compute-plane-services/nvca/pkg/featureflag/featureflag_workload.go
  • src/compute-plane-services/nvca/pkg/featureflag/featureflag_workload_test.go
  • src/compute-plane-services/nvca/pkg/nvca/k8scomputebackend.go
  • src/compute-plane-services/nvca/pkg/nvca/k8scomputebackend_miniservice.go
  • src/compute-plane-services/nvca/pkg/storage/modelcache.go
🚧 Files skipped from review as they are similar to previous changes (10)
  • src/compute-plane-services/nvca/internal/miniservice/status_test.go
  • src/compute-plane-services/nvca/pkg/featureflag/BUILD.bazel
  • src/compute-plane-services/nvca/pkg/storage/modelcache.go
  • src/compute-plane-services/nvca/pkg/apis/nvca/v1alpha1/miniservice_status_types.go
  • src/compute-plane-services/nvca/internal/miniservice/revision.go
  • src/compute-plane-services/nvca/internal/util/k8sutil/pod_test.go
  • src/compute-plane-services/nvca/pkg/nvca/k8scomputebackend_miniservice.go
  • src/compute-plane-services/nvca/internal/util/k8sutil/pod.go
  • src/compute-plane-services/nvca/pkg/nvca/k8scomputebackend.go
  • src/compute-plane-services/nvca/internal/miniservice/status.go

Comment thread src/compute-plane-services/nvca/internal/miniservice/reconcile.go
Comment thread src/compute-plane-services/nvca/internal/miniservice/reconcile.go
Comment thread src/compute-plane-services/nvca/pkg/apis/nvca/v1alpha1/BUILD.bazel Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/compute-plane-services/nvca/internal/miniservice/status.go`:
- Around line 507-510: Update the reconciliation flow around
getFunctionWorkerContainerStatus so a new or unscheduled pod with no container
statuses is treated as pending rather than terminally failed. Either return and
handle a found/not-found indicator, or defer the lookup while
utilsStatus.Scheduling is true, while preserving the existing scheduling/pending
branch and only propagating genuine status errors.
- Around line 530-551: Update the worker degradation handling around the
AdditionalPodStatus checks so timeout-only Always pods also append
MiniServiceStatusReasonDegradedWorker when the worker container is affected. Do
not rely solely on aps.Degraded.Containers, since IsPodDegraded omits those
entries for RestartPolicy Always; reuse the existing worker-container
identification and preserve the existing handling for other degradation reasons.

In `@src/compute-plane-services/nvca/pkg/nvca/k8scomputebackend_miniservice.go`:
- Around line 355-365: Guard the ImagePullIssueTotal update in the multi-issue
loop with the same nil check used elsewhere for the metrics context. Update the
block around ImagePullIssuesReported so
metrics.ImagePullIssueTotal.WithLabelValues(...).Inc() executes only when
metrics is non-nil, while preserving registry deduplication and insertion
behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 79f49539-00d8-4dcc-99a7-33e26e9ecd36

📥 Commits

Reviewing files that changed from the base of the PR and between 8c29476 and a5c2d1b.

⛔ Files ignored due to path filters (1)
  • src/compute-plane-services/nvca/pkg/apis/nvca/v1alpha1/zz_generated.deepcopy.go is excluded by !**/zz_generated.*
📒 Files selected for processing (19)
  • src/compute-plane-services/nvca/internal/miniservice/controller.go
  • src/compute-plane-services/nvca/internal/miniservice/reconcile.go
  • src/compute-plane-services/nvca/internal/miniservice/reconcile_test.go
  • src/compute-plane-services/nvca/internal/miniservice/revision.go
  • src/compute-plane-services/nvca/internal/miniservice/status.go
  • src/compute-plane-services/nvca/internal/miniservice/status_karta_test.go
  • src/compute-plane-services/nvca/internal/miniservice/status_test.go
  • src/compute-plane-services/nvca/internal/util/k8sutil/pod.go
  • src/compute-plane-services/nvca/internal/util/k8sutil/pod_test.go
  • src/compute-plane-services/nvca/pkg/apis/nvca/v1alpha1/BUILD.bazel
  • src/compute-plane-services/nvca/pkg/apis/nvca/v1alpha1/generated.openapi.go
  • src/compute-plane-services/nvca/pkg/apis/nvca/v1alpha1/miniservice_status_types.go
  • src/compute-plane-services/nvca/pkg/apis/nvca/v1alpha1/miniservice_types.go
  • src/compute-plane-services/nvca/pkg/featureflag/BUILD.bazel
  • src/compute-plane-services/nvca/pkg/featureflag/featureflag_workload.go
  • src/compute-plane-services/nvca/pkg/featureflag/featureflag_workload_test.go
  • src/compute-plane-services/nvca/pkg/nvca/k8scomputebackend.go
  • src/compute-plane-services/nvca/pkg/nvca/k8scomputebackend_miniservice.go
  • src/compute-plane-services/nvca/pkg/storage/modelcache.go
🚧 Files skipped from review as they are similar to previous changes (16)
  • src/compute-plane-services/nvca/internal/miniservice/revision.go
  • src/compute-plane-services/nvca/internal/miniservice/status_test.go
  • src/compute-plane-services/nvca/pkg/apis/nvca/v1alpha1/miniservice_types.go
  • src/compute-plane-services/nvca/pkg/apis/nvca/v1alpha1/BUILD.bazel
  • src/compute-plane-services/nvca/internal/util/k8sutil/pod_test.go
  • src/compute-plane-services/nvca/internal/miniservice/controller.go
  • src/compute-plane-services/nvca/pkg/apis/nvca/v1alpha1/generated.openapi.go
  • src/compute-plane-services/nvca/pkg/storage/modelcache.go
  • src/compute-plane-services/nvca/pkg/apis/nvca/v1alpha1/miniservice_status_types.go
  • src/compute-plane-services/nvca/pkg/featureflag/featureflag_workload_test.go
  • src/compute-plane-services/nvca/internal/miniservice/status_karta_test.go
  • src/compute-plane-services/nvca/pkg/featureflag/featureflag_workload.go
  • src/compute-plane-services/nvca/pkg/nvca/k8scomputebackend.go
  • src/compute-plane-services/nvca/internal/miniservice/reconcile_test.go
  • src/compute-plane-services/nvca/internal/util/k8sutil/pod.go
  • src/compute-plane-services/nvca/internal/miniservice/reconcile.go

Comment thread src/compute-plane-services/nvca/internal/miniservice/status.go Outdated
Comment thread src/compute-plane-services/nvca/internal/miniservice/status.go

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
src/compute-plane-services/nvca/pkg/featureflag/featureflag_workload.go (1)

67-71: 🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win

Handle YAML null before ranging cfg.FeatureFlags. A top-level null in config.yaml leaves cfg nil here, so cfg.FeatureFlags can panic during reconciliation. Unmarshal into a value or return early when cfg == nil; add a regression test.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/compute-plane-services/nvca/pkg/featureflag/featureflag_workload.go`
around lines 67 - 71, Prevent nil dereferencing after yaml.Unmarshal in the
feature-flag reconciliation flow by handling a top-level null before ranging
cfg.FeatureFlags. Update the surrounding function to return an empty result or
otherwise preserve its existing no-configuration behavior when cfg is nil, and
add a regression test covering a YAML null configuration.

Source: MCP tools

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@src/compute-plane-services/nvca/pkg/featureflag/featureflag_workload.go`:
- Around line 67-71: Prevent nil dereferencing after yaml.Unmarshal in the
feature-flag reconciliation flow by handling a top-level null before ranging
cfg.FeatureFlags. Update the surrounding function to return an empty result or
otherwise preserve its existing no-configuration behavior when cfg is nil, and
add a regression test covering a YAML null configuration.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 46eb1379-5640-4008-bdbb-8497a770e2b5

📥 Commits

Reviewing files that changed from the base of the PR and between a5c2d1b and a4def10.

📒 Files selected for processing (6)
  • src/compute-plane-services/nvca/internal/miniservice/BUILD.bazel
  • src/compute-plane-services/nvca/internal/miniservice/reconcile.go
  • src/compute-plane-services/nvca/pkg/featureflag/BUILD.bazel
  • src/compute-plane-services/nvca/pkg/featureflag/featureflag_workload.go
  • src/compute-plane-services/nvca/pkg/featureflag/featureflag_workload_test.go
  • src/compute-plane-services/nvca/pkg/nvca/k8scomputebackend.go
🚧 Files skipped from review as they are similar to previous changes (4)
  • src/compute-plane-services/nvca/pkg/featureflag/BUILD.bazel
  • src/compute-plane-services/nvca/pkg/featureflag/featureflag_workload_test.go
  • src/compute-plane-services/nvca/internal/miniservice/reconcile.go
  • src/compute-plane-services/nvca/pkg/nvca/k8scomputebackend.go

@estroz
estroz enabled auto-merge July 23, 2026 05:13
@estroz
estroz disabled auto-merge July 23, 2026 05:40
@estroz
estroz force-pushed the estroczynski/feat/worker-health branch from f4a0b50 to 8e4310b Compare July 23, 2026 05:41
@estroz estroz changed the title [Draft] feat(nvca): add worker-based health per-instance feature feat(nvca): add worker-based health per-instance feature Jul 23, 2026
@estroz
estroz added this pull request to the merge queue Jul 23, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants