Skip to content

CRD drift: gate rendered output against the committed crd.yaml (keep's spec.backup was silently pruned in-cluster) #5

Open

Description

@chrischeng-c4

Capability Alignment

Kubernetes-native deployment: an operator's committed CRD must be the exact
projection of its Rust spec type. When it is not, the API server silently
prunes the missing fields and the operator behaves as if the user never set
them — a class of failure with no error message anywhere in the stack.

Scope

Every app whose CLI ships k8s crd render (tape, relay, keep, defer, lumen,
beam as they gain one) gets a test that renders the CRD from the Rust source
and asserts byte equality with the committed k8s/operator/crd.yaml, failing
with the exact regeneration command in its message.

Explicitly in scope:

  • The shared test helper, so each app adds a one-liner rather than its own
    bespoke comparison.
  • An audit pass over every existing committed CRD to find the drift already
    present (keep's is proven below; the others are unaudited).
  • keep's spec.backup in particular: confirm the backup CronJob renders
    in-cluster once the CRD carries the field.

Explicitly out of scope: Dockerfile render drift (faberline/workspace#2 owns that), and any
change to what the CRDs contain beyond closing measured drift.

Acceptance Criteria

  1. cargo test -p <app> fails when crd.yaml is stale, for every app with a
    k8s crd render command.
  2. The failure message names the exact command that regenerates the file.
  3. Every committed CRD in the repo is byte-identical to its generator output
    at the close of this work item.
  4. A Keep CR with spec.backup set survives a real apply and produces the
    backup CronJob — proving the pruning consequence is actually closed, not
    just the file diff.

Reference Context

Found while regenerating keep's CRD for faberline/core#14 (commit fba67420a4).

git show HEAD~1:apps/keep/k8s/operator/crd.yaml | grep -c backup returned
0, while KeepSpec.backup has existed in Rust since 4e22bde50d (faberline/axiom#1249).
v1 CRDs prune unknown fields by default, so spec.backup: on a Keep CR was
being dropped by the API server and the backup CronJob never rendered. The
field was present in Rust, absent from the cluster's schema, and nothing in
CI compared the two.

keep's tests do call crd_yaml() — they assert on its content in isolation,
which is exactly the shape of test that passes while the committed artifact
rots. The regeneration in fba67420a4 also swept in description changes for
replicasPerShard and resources, so the drift was not limited to one
field.

relay's CRD, by contrast, was current — a clean 5-line addition. So this is
per-app rot, not a systemic generator problem, which is precisely why a gate
is the fix rather than a one-time cleanup.

Activity

  1. chrischeng-c4 commented on Jul 24, 2026

    @chrischeng-c4
    ContributorAuthor

    Survey caveat found after filing: two apps would gate vacuously

    Verified while filing the faberline/core#14 follow-ups. Not every k8s crd render is a
    generator, so a naive "render and diff against the committed file" gate would
    pass trivially in two places by comparing the file against itself:

    app crd render implementation gate meaningful?
    keep Rust KeepSpec -> crd_yaml() yes
    relay Rust RelaySpec -> crd_yaml() yes
    tape Rust TapeSpec -> crd_yaml() yes
    lumen Rust spec -> crd_yaml() yes
    defer Rust DeferSpec, behind --features operator; default build falls back to include_str! (src/bin/defer.rs:988) only with the feature on
    beam strip_ownership_markers(include_str!(...)), no spec type at all (src/dx.rs:105) no — vacuous

    Two consequences for this work item:

    1. defer's gate must build with --features operator, or it asserts nothing.
      A gate that silently degrades to a tautology when a feature flag is off is
      worse than no gate, so it should fail loudly rather than skip.
    2. beam cannot have a real gate until it has a real BeamSpec — filed as
      beam: the k8s operator surface is a placeholder — build it or withdraw the claim beam#3. Until then beam should be explicitly excluded with a comment
      pointing at beam: the k8s operator surface is a placeholder — build it or withdraw the claim beam#3, not quietly included and passing.

    This is also a reminder that the shared helper should assert on the
    generator, not on whatever crd render happens to print.

  2. chrischeng-c4 commented on Oct 23, 2026

    @chrischeng-c4
    ContributorAuthor

    issue-plan: set to priority:p3 so the work step does not duplicate it. The same work is already submitted: #14 (PR #16, gate committed k8s/operator/crd.yaml against the generator, open and waiting on CI) and PR #8 (claude/keep-5-crd-drift, drift test + regenerated CRD + backup CronJob, open). Close this issue once those merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions