feat: capture checkpoints via Snapshot/SnapshotContent (Component C)

[Ronkahn21]

Overview:

Component C of the Snapshot/SnapshotContent migration: move container-checkpoint capture onto the new CRDs using a CSI VolumeSnapshot / external-snapshotter split. Restore is untouched (stays on DynamoCheckpoint + pod annotations). External DGD API is unchanged.

Details:

• Operator SnapshotReconciler (new, cluster-wide) ≈ snapshot-controller: creates the cluster-scoped SnapshotContent work order (source podRef{name,uid} + nodeName + checkpointID/artifactVersion metadata), binds Snapshot.status.boundSnapshotContentName, mirrors SnapshotContent.status → Snapshot.status, and owns finalizers on both. Cluster→namespaced ownerRef is impossible, so the link is spec.snapshotRef + a finalizer; the status-mirror mapper unwraps cache.DeletedFinalStateUnknown so deletes don't wedge.
• DynamoCheckpoint controller: Owns(&Snapshot{}) (update-only predicate); completion reads Snapshot.status; deadline + Job-failure hang guards.
• Agent SnapshotContentReconciler (new, per-node, action-only) ≈ CSI driver: controller-runtime manager with a node-label-scoped cache; dumps via a NodeCheckpointer interface; writes only SnapshotContent.status. UID provenance check, inFlight guard + lease-renewal goroutine. The capture-side pod informer + reconcileCheckpointPod are removed (the work order is the sole trigger; no double-dump).
• API: SnapshotContent.spec.source = {podRef{name,uid}, nodeName} (both required); storage coords ride as existing label/annotation; protocol.ObserveCheckpointJob/observation types removed.

Verification: operator controller envtest + api tests pass (envtest run outside sandbox); agent internal/controller + protocol tests pass in a Linux container; both modules build; codegen has zero drift.

Where should the reviewer start?

• deploy/operator/internal/controller/snapshot_reconciler.go
• deploy/operator/internal/controller/dynamocheckpoint_controller.go (observeSnapshot)
• deploy/snapshot/internal/controller/snapshotcontent_reconciler.go

Related Issues

🚫 This PR is NOT linked to an issue:

• Confirmed — no related issue (intermediate component PR into the feat/snapshot-crd fork base; tracked in the migration's internal state doc — upstream issue is created at the base→upstream PR)

[Ronkahn21]

Addressed the review comments in 65208a9 + 3e95529. Per-comment:

dynamocheckpoint_controller.go

• ptr.To/new → CreatedAt = ptr.To(metav1.Now()) in the new markCheckpointReady.
• IsSnapshotSucceeded util + move update out → added IsSnapshotSucceeded/IsSnapshotFailed in api/v1alpha1; observeSnapshot uses them and the status write moved into markCheckpointReady.
• mixes observe + status update → split: observeSnapshot only decides; markCheckpointReady/failCreating write.
• job status should expose jobFailed → checkpointJobFailed(job) reports the failed condition; checkpoint fails when the Job failed and the Snapshot has not succeeded.
• also deleted should be watched → Owns(&Snapshot{}) predicate DeleteFunc -> true.
• use SSA → kept Status().Update here. The other ~8 status writes in this controller use Update; mixing SSA on the same status subresource risks silently dropping conditions (e.g. JobCreated) and is not exercised by the fake-client/envtest suite. Can do a full SSA-status migration as a separate follow-up if you want it.

snapshot_reconciler.go

• separate get + validate → getSourcePod + validateSourcePod.
• follow ensureSnapshotContent get-or-create / fold findBoundContent in → ensureSnapshotContent now get-or-creates (Get, else build + SSA) and returns the content, mirroring ensureSnapshot in checkpoint_snapshot.go.
• get once, not in every function → Reconcile resolves the content once and passes it to propagateStatus; no re-Get.
• status as a separate step / call it propagateStatus → renamed bindAndMirror -> propagateStatus, now a separate reconcile step taking the content.
• delete without pre-Get, block to wait → handleDelete deletes by name with IgnoreNotFound (no pre-Get), then requeues until a Get confirms it is gone before dropping the Snapshot finalizer.
• start without finalizer on the SnapshotContent → removed the finalizer from buildSnapshotContent.
• start without the predicate → removed the Watches(&SnapshotContent{}) predicate; kept the mapper + cache.DeletedFinalStateUnknown tombstone unwrap.

Verified: go build/go vet clean, go test ./api/... + controller envtest (64 specs + table tests) pass, agent module untouched, no codegen drift.

Note: per-thread replies were blocked by an existing pending review on this PR.