Skip to content

feat(flux): bootstrap verified OCI source #2164

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jfroy wants to merge 1 commit into from
Open

feat(flux): bootstrap verified OCI source #2164

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .github/workflows/cluster-release.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ permissions:
contents: read

env:
# Must match status.webhookPath for Receiver cluster-release/flux-system.
FLUX_RECEIVER_URL: https://flux-webhook.kantai.xyz/hook/ef1ac25eeff8057bf82070418a3eabb755c4821b29ff32377c75863bc1692ab5
Comment on lines +22 to +23

@claude claude Bot Aug 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This digest can't be validated from Git, and this PR changes both inputs that feed it.

status.webhookPath is computed by notification-controller from the Receiver object itself, and this PR renames it (github-webhook → cluster-release) and removes secretRef — so the path will not be whatever the currently-deployed github-webhook Receiver reports, and it isn't the sha256 of the old token either. git log -S shows this hash is new in this PR, so it wasn't read off a live object.

Please confirm it against the live object after the Receiver reconciles, before relying on it:

kubectl -n flux-system get receiver cluster-release -o jsonpath='{.status.webhookPath}'

Two follow-ons worth noting since the value is a literal:

  • notification-controller derives the path partly from Receiver-instance identity, so any prune-and-recreate of Receiver/cluster-release (namespace rebuild, moving the manifest between Kustomizations, spec immutable-field churn) rotates the path and silently breaks releases with no signal in Git. A PrometheusRule on the workflow failing, or a kubectl lookup at the start of the Notify Flux step, would catch it.
  • there is no Receiver/github-webhook → cluster-release migration for the GitHub-side webhook. The repo/org webhook still points at the old /hook/<old-digest> path and will start returning 404 on every push once the old Receiver is pruned. It should be deleted in the GitHub UI as part of this rollout.

Comment on lines +22 to +23

@claude claude Bot Aug 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This literal path can't be right yet, and it's the one thing here that can't be validated before merge.

status.webhookPath is generated by notification-controller as a SHA-256 digest that includes the Receiver's UID (plus the token, empty here). This PR renames the Receiver — github-webhook → cluster-release — so kustomize-controller will create a brand-new object with a brand-new UID, and its path will not be whatever was read out of the cluster to produce this value. The only way this digest is already correct is if Receiver/cluster-release was applied to the cluster out-of-band and Flux then adopts that same object (same UID); if it ever gets pruned and recreated, the path silently changes again.

Failure mode is quiet: the merge run publishes and promotes the artifact fine, then Notify Flux POSTs to a path notification-controller doesn't serve, gets 404 for 30 attempts, and fails the job after 5 minutes — while the cluster keeps running old state until the 1h Git interval fires.

Please confirm after this reconciles:

kubectl -n flux-system get receiver cluster-release -o jsonpath='{.status.webhookPath}'

And consider moving the URL out of the tree into a repository variable (vars.FLUX_RECEIVER_URL), so a Receiver recreation is a settings change rather than a commit that re-triggers the very workflow it's fixing.

OCI_REPOSITORY: ghcr.io/${{ github.repository }}/cluster

jobs:
Expand Down Expand Up @@ -103,3 +105,21 @@ jobs:
DIGEST: ${{ steps.publish.outputs.digest }}
run: |
flux tag artifact "oci://${OCI_REPOSITORY}@${DIGEST}" --tag latest

- name: Notify Flux
run: |
token="$(curl --fail-with-body --silent --show-error \
--header "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" \
"${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=notification-controller" \
| jq -er '.value')"
for attempt in $(seq 1 30); do
if curl --fail-with-body --silent --show-error \
--request POST \
--header "Authorization: Bearer ${token}" \
"${FLUX_RECEIVER_URL}"; then
exit 0
fi
echo "Flux Receiver unavailable (attempt ${attempt}/30); retrying in 10 seconds"
sleep 10
Comment on lines +115 to +123

@claude claude Bot Aug 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The retry loop treats every non-2xx the same, so a permanent rejection burns the full 5 minutes and then fails with no useful signal. That is the likely failure mode here: if any of the three CEL validations doesn't match, notification-controller returns 403, and if the hardcoded path drifts from status.webhookPath it returns 404 forever. Both are indistinguishable from "Receiver not rolled out yet".

Consider capturing the status code and only retrying on connection errors / 5xx (and 404 for the first few attempts), failing fast and loudly on 403:

code="$(curl --silent --output /dev/stderr --write-out '%{http_code}' \
  --request POST --header "Authorization: Bearer ${token}" "${FLUX_RECEIVER_URL}")"
case "${code}" in
  2*) exit 0 ;;
  403) echo "Receiver rejected the OIDC token (CEL validation failed)"; exit 1 ;;
esac

done
Comment on lines +111 to +124

@claude claude Bot Aug 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ID token is minted once, outside a loop that can run for 5 minutes. GitHub Actions OIDC tokens are short-lived (exp - iat is on the order of minutes), so in exactly the scenario this retry exists for — Receiver not yet reconciled — the later attempts can start failing 401 on an expired token rather than 404 on a missing path, and the log line will still say "Flux Receiver unavailable". Minting inside the loop is cheap and removes the ambiguity:

Suggested change
token="$(curl --fail-with-body --silent --show-error \
--header "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" \
"${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=notification-controller" \
| jq -er '.value')"
for attempt in $(seq 1 30); do
if curl --fail-with-body --silent --show-error \
--request POST \
--header "Authorization: Bearer ${token}" \
"${FLUX_RECEIVER_URL}"; then
exit 0
fi
echo "Flux Receiver unavailable (attempt ${attempt}/30); retrying in 10 seconds"
sleep 10
done
run: |
get_token() {
curl --fail-with-body --silent --show-error \
--header "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" \
"${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=notification-controller" \
| jq -er '.value'
}
for attempt in $(seq 1 30); do
token="$(get_token)"
if curl --fail-with-body --silent --show-error \
--request POST \
--header "Authorization: Bearer ${token}" \
"${FLUX_RECEIVER_URL}"; then
exit 0
fi
echo "Flux Receiver unavailable (attempt ${attempt}/30); retrying in 10 seconds"
sleep 10
done

Separately: a 403 from a failed CEL validation is permanent, and retrying it 30 times buys nothing but 5 minutes of confusing logs. Worth breaking out of the loop on 4xx-other-than-404 if you want the misconfiguration to surface fast.

Comment on lines +111 to +124

@claude claude Bot Aug 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The OIDC token is minted once, outside a loop that can run for ~5 minutes (30 × 10s). GitHub Actions ID tokens are short-lived, so in exactly the scenario this retry exists for — a Receiver that stays unavailable through the initial rollout — the later attempts can start failing on an expired exp rather than on the Receiver, turning a recoverable wait into a permanent failure with a misleading Flux Receiver unavailable message. Minting per attempt is cheap and removes the coupling between the retry budget and the token lifetime.

Suggested change
token="$(curl --fail-with-body --silent --show-error \
--header "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" \
"${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=notification-controller" \
| jq -er '.value')"
for attempt in $(seq 1 30); do
if curl --fail-with-body --silent --show-error \
--request POST \
--header "Authorization: Bearer ${token}" \
"${FLUX_RECEIVER_URL}"; then
exit 0
fi
echo "Flux Receiver unavailable (attempt ${attempt}/30); retrying in 10 seconds"
sleep 10
done
for attempt in $(seq 1 30); do
token="$(curl --fail-with-body --silent --show-error \
--header "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" \
"${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=notification-controller" \
| jq -er '.value')"
if curl --fail-with-body --silent --show-error \
--request POST \
--header "Authorization: Bearer ${token}" \
"${FLUX_RECEIVER_URL}"; then
exit 0
fi
if [[ "${attempt}" -lt 30 ]]; then
echo "Flux Receiver unavailable (attempt ${attempt}/30); retrying in 10 seconds"
sleep 10
fi
done
exit 1

(The -lt 30 guard also drops the pointless 10s sleep before the final exit 1.)

Separately, worth considering: the loop retries indiscriminately, so a genuine misconfiguration — a CEL validation that never matches, or a stale FLUX_RECEIVER_URL — burns the full five minutes as 401/404 before failing. A fast-fail on those statuses would make a broken Receiver config much more obvious than a wall of "unavailable" lines.

exit 1
Comment on lines +115 to +125

@claude claude Bot Aug 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The retry loop can't distinguish "Receiver not rolled out yet" from "permanently misconfigured", so both cost 5 minutes and produce the same misleading log line.

--fail-with-body makes curl fail on any 4xx/5xx. A wrong FLUX_RECEIVER_URL digest → 404, a rejected OIDC token (bad audience, or a validations expression that doesn't match) → 401/403. None of those become truthy on retry, but each burns 30 attempts and prints Flux Receiver unavailable, which points the reader at the cluster rather than at the token or the URL.

Capturing the status code lets you retry only what's actually transient and surface the CEL rejection message from the Receiver on the terminal cases:

Suggested change
for attempt in $(seq 1 30); do
if curl --fail-with-body --silent --show-error \
--request POST \
--header "Authorization: Bearer ${token}" \
"${FLUX_RECEIVER_URL}"; then
exit 0
fi
echo "Flux Receiver unavailable (attempt ${attempt}/30); retrying in 10 seconds"
sleep 10
done
exit 1
for attempt in $(seq 1 30); do
body="$(mktemp)"
code="$(curl --silent --show-error \
--output "${body}" --write-out '%{http_code}' \
--request POST \
--header "Authorization: Bearer ${token}" \
"${FLUX_RECEIVER_URL}")"
case "${code}" in
2*)
exit 0
;;
404|502|503|504)
echo "Flux Receiver unavailable (HTTP ${code}, attempt ${attempt}/30); retrying in 10 seconds"
;;
*)
echo "Flux Receiver rejected the notification (HTTP ${code}):"
cat "${body}"
exit 1
;;
esac
sleep 10
done
exit 1

Note that 404 stays in the retry set on purpose — notification-controller returns 404 both for an unknown path and for a Receiver it hasn't indexed yet, so it's ambiguous. That makes the status.webhookPath check in my other comment the thing that actually distinguishes them.

Comment on lines +111 to +125

@claude claude Bot Aug 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The OIDC token is minted once, outside a loop that can run for 5 minutes (30 × 10s). GitHub Actions ID tokens are short-lived, so the later attempts can start failing on an expired exp rather than on the Receiver being unavailable — and since every non-2xx is treated identically, the step would burn the full window and then fail for a reason the log message misattributes.

Minting inside the loop keeps the retry meaningful:

Suggested change
token="$(curl --fail-with-body --silent --show-error \
--header "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" \
"${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=notification-controller" \
| jq -er '.value')"
for attempt in $(seq 1 30); do
if curl --fail-with-body --silent --show-error \
--request POST \
--header "Authorization: Bearer ${token}" \
"${FLUX_RECEIVER_URL}"; then
exit 0
fi
echo "Flux Receiver unavailable (attempt ${attempt}/30); retrying in 10 seconds"
sleep 10
done
exit 1
- name: Notify Flux
run: |
for attempt in $(seq 1 30); do
token="$(curl --fail-with-body --silent --show-error \
--header "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" \
"${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=notification-controller" \
| jq -er '.value')"
if curl --fail-with-body --silent --show-error \
--request POST \
--header "Authorization: Bearer ${token}" \
"${FLUX_RECEIVER_URL}"; then
exit 0
fi
echo "Flux Receiver unavailable (attempt ${attempt}/30); retrying in 10 seconds"
sleep 10
done
exit 1

Comment on lines +111 to +125

@claude claude Bot Aug 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two issues with the retry loop:

  1. The ID token is minted once, outside the loop, but the loop spans ~5 minutes (30 × 10s). GitHub Actions OIDC tokens are short-lived, and notification-controller validates exp, so the later attempts can fail on token expiry rather than on Receiver availability — i.e. the retry never actually helps in the tail of the window. Move the token request inside the loop.

  2. It retries indiscriminately. A permanent rejection — a CEL validation failure (403) from a claim mismatch, or a 404 from a webhook-path drift — burns the full 5 minutes before failing, which reads as "Receiver unavailable" rather than "misconfigured". Capturing the HTTP status (--write-out '%{http_code}' / -o /dev/null) and bailing out on 4xx other than 404-during-rollout would make the failure mode legible.

18 changes: 0 additions & 18 deletions kubernetes/apps/flux-system/instance/ks/externalsecret.yaml
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion kubernetes/apps/flux-system/instance/ks/httproute.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: github-webhook
name: cluster-release
annotations:
gatus.home-operations.com/endpoint: |-
conditions: ["[STATUS] == 404"]
Expand Down
1 change: 0 additions & 1 deletion kubernetes/apps/flux-system/instance/ks/kustomization.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ./externalsecret.yaml
- ./grafanadashboard.yaml
- ./helmrelease.yaml
- ./httproute.yaml
Expand Down
24 changes: 13 additions & 11 deletions kubernetes/apps/flux-system/instance/ks/receiver.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,20 @@
apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
name: github-webhook
name: cluster-release
spec:
type: github
events:
- ping
- push
secretRef:
name: github-webhook-token-secret
type: generic-oidc
oidcProviders:
- issuerURL: https://token.actions.githubusercontent.com
audience: notification-controller
validations:
- expression: claims.repository == "jfroy/flatops"
message: unexpected repository
- expression: claims.ref == "refs/heads/main"
message: release must run from main
- expression: claims.workflow_ref == "jfroy/flatops/.github/workflows/cluster-release.yaml@refs/heads/main"
message: unexpected workflow
resources:
- apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
name: flux-system
- apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
kind: OCIRepository
name: flux-system
Comment on lines 19 to 22

@claude claude Bot Aug 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropping GitRepository/flux-system (and Kustomization/flux-system) from resources leaves the cluster with no push-triggered reconciliation at all for the duration of the stack.

After this PR:

  • Receiver/github-webhook is gone, so the webhook path registered in GitHub repo settings (sha256(github-webhook + flux-system + token)) 404s, and the new Receiver can't accept it anyway — it's generic-oidc, not github/HMAC.
  • The only resource this Receiver pokes is OCIRepository/flux-system, which nothing consumes yet: kubernetes/cluster/ks.yaml (cluster-apps, cluster-vap) and the FluxInstance sync (kind: GitRepository, interval: 1h) all still source from Git.

So from merge until stack step 4 lands, every change to main waits on the 1h sync interval plus each child Kustomization's own 1h interval instead of reconciling in seconds.

Cheap fix — keep the Git source in the list until the source migration completes, then remove it:

Suggested change
resources:
- apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
name: flux-system
- apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
kind: OCIRepository
name: flux-system
resources:
- apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
name: flux-system
# TODO: drop once the sync source migrates to OCIRepository (stack step 4).
- apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
name: flux-system

Comment on lines 19 to 22

@claude claude Bot Aug 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This drops push-triggered reconciliation of the Git source, and nothing consumes OCIRepository/flux-system yet.

Every Kustomization in the repo still has sourceRef: {kind: GitRepository, name: flux-system} — cluster-apps and cluster-vap (kubernetes/cluster/ks.yaml), flux-instance, and the FluxInstance sync itself (sync.kind: GitRepository, interval: 1h in instance/ks/helm-values.yaml). The old github-webhook Receiver poked GitRepository/flux-system and Kustomization/flux-system; the new one pokes only the OCI source. So from the moment this merges until PRs 3 and 4 land, every merge to main waits on the 1h GitRepository interval.

That also creates a bootstrap gap for this PR itself: Receiver/cluster-release only exists after flux-instance reconciles the renamed manifest, which is now up to an hour after merge — well past the 5-minute retry window in the workflow. The Notify Flux step on this PR's own merge commit will almost certainly exhaust its 30 attempts unless you flux reconcile source git flux-system && flux reconcile ks flux-instance manually.

Since the release workflow runs on push to main, keeping the Git resources in the trigger list restores the old behaviour for free and can be removed in PR 4:

Suggested change
resources:
- apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
name: flux-system
- apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
kind: OCIRepository
name: flux-system
resources:
- apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
name: flux-system
# Transitional: most Kustomizations still source from Git until the
# source migration completes. Remove once the migration lands.
- apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
name: flux-system
- apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
name: flux-system

Comment on lines 19 to 22

@claude claude Bot Aug 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropping GitRepository/flux-system (and Kustomization/flux-system) from resources leaves the Git source with no notification path for the duration of the migration.

Everything still consuming Git today — the FluxInstance sync GitRepository (interval: 1h), cluster-apps, cluster-vap, flux-instance — only picks up a merge on its 1h poll once the old github Receiver is pruned. That's the window in which stack PRs 3 and 4 land, so those two merges go from near-instant to up-to-an-hour reconcile unless you flux reconcile kustomization cluster-apps --with-source by hand each time.

Cheap fix while the stack is in flight — the release workflow already fires on every kubernetes/** push to main, so one notification can poke both sources:

Suggested change
resources:
- apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
name: flux-system
- apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
kind: OCIRepository
name: flux-system
resources:
- apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
name: flux-system
- apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
name: flux-system

Then drop the GitRepository entry in PR 4 when the Git source goes away.

Comment on lines 19 to 22

@claude claude Bot Aug 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This drops push-triggered reconciliation for the Git source, which nothing consumes the OCI source yet to replace.

After this merges, OCIRepository/flux-system is the only notified resource, but FluxInstance.spec.sync is still kind: GitRepository (instance/ks/helm-values.yaml) and every Kustomization — cluster-vap, cluster-apps, and all app ks.yaml — still has sourceRef.kind: GitRepository. The old github-webhook Receiver and its path are gone, so for the duration of PRs 3–4 a push to main only reconciles at interval: 1h instead of within seconds.

Since the Cluster Release workflow already fires on the same push: main + kubernetes/** trigger, adding the Git source to this Receiver restores that for free until the migration completes:

Suggested change
resources:
- apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
name: flux-system
- apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
kind: OCIRepository
name: flux-system
resources:
- apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
name: flux-system
- apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
name: flux-system

Comment on lines 19 to 22

@claude claude Bot Aug 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This replaces the push trigger before its replacement is consumed by anything. After this PR, the only Receiver in the repo notifies OCIRepository/flux-system — but nothing sources from it yet: cluster-apps and cluster-vap (kubernetes/cluster/ks.yaml), flux-instance (kubernetes/apps/flux-system/instance/ks.yaml), and every other ks.yaml still use sourceRef.kind: GitRepository.

Net effect for the window between this PR and stack step 4:

  • GitRepository/flux-system and Kustomization/flux-system lose their webhook, so a push to main reconciles only on the FluxInstance sync interval: 1h (ks/helm-values.yaml).
  • The GitHub repo webhook still points at the old /hook/<sha256(github-webhook+flux-system+token)> path, which no longer exists → 404s on every delivery.

Since Receivers are additive, consider keeping the github-webhook Receiver (and its externalsecret.yaml) in place alongside this one and deleting both in step 4, once the Kustomizations actually consume the OCI source.

Comment on lines 19 to 22

@claude claude Bot Aug 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropping GitRepository/flux-system here leaves the cluster with no push-triggered git reconciliation for the whole duration of the migration.

Every ks.yaml in the repo — plus the FluxInstance-generated flux-system Kustomization itself (helm-values.yaml: sync.kind: GitRepository, interval: 1h) — still resolves sourceRef to GitRepository/flux-system. Steps 3 and 4 of the stack are what move those over. Until then, the only thing that poked the git source was the old github-webhook Receiver, and this PR deletes it, so the GitHub push webhook configured in repo settings starts hitting an unknown /hook/<old-hash> path (404). Net effect after merge: kubernetes/** changes land on the cluster up to an hour late, and the new Receiver only refreshes an OCIRepository nothing consumes yet.

Since the release workflow runs on every push to main that touches kubernetes/**, listing both sources keeps the fast path alive through the migration window:

Suggested change
resources:
- apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
name: flux-system
- apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
kind: OCIRepository
name: flux-system
resources:
- apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
name: flux-system
- apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
name: flux-system

The GitRepository entry can then be removed in step 4 along with the git source itself.

20 changes: 20 additions & 0 deletions kubernetes/cluster/ocirepository.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
---
# yaml-language-server: $schema=https://crd.kantai.xyz/source.toolkit.fluxcd.io/ocirepository_v1.json
apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
name: flux-system
namespace: flux-system
annotations:
# Keep the source in place when FluxInstance takes over its management.
kustomize.toolkit.fluxcd.io/prune: Disabled

@claude claude Bot Aug 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Disabled is capitalized here. Flux's documented value for this annotation is disabled, and kustomize-controller passes kustomizev1.DisabledValue ("disabled") as an exact-match exclusion when computing the prune set — every other occurrence in this repo uses lowercase (kubernetes/components/common/kustomization.yaml:15, the eight objectbucketclaim.yaml files).

If the match is case-sensitive, this annotation is a no-op, which defeats its stated purpose: when PR 4 deletes this file, cluster-apps/root flux-system will garbage-collect OCIRepository/flux-system — the source the root Kustomization is by then reading from — instead of leaving it for FluxInstance to adopt.

Suggested change
kustomize.toolkit.fluxcd.io/prune: Disabled
kustomize.toolkit.fluxcd.io/prune: disabled

spec:
interval: 5m

@claude claude Bot Aug 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

interval: 5m deviates from every other OCIRepository in this repo (app-template, flux-instance, flux-operator all use 1h) and from the FluxInstance sync interval. Because ref.tag: latest is mutable, this re-resolves the tag against GHCR 12×/hour, and each resolution runs keyless cosign verification against Fulcio/Rekor. The Receiver added in this PR is the push path, so the poll interval is only a fallback — 1h matches convention and keeps sigstore traffic down.

Suggested change
interval: 5m
interval: 1h

ref:
tag: latest
url: oci://ghcr.io/jfroy/flatops/cluster
Comment on lines +8 to +15

@claude claude Bot Aug 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things to confirm on this source, both of which the rollout gate would catch but are cheaper to check up front:

No pull credentials. There's no secretRef / serviceAccountName / provider, and there is no registry secret anywhere under kubernetes/apps/flux-system/ — every other OCIRepository in the repo targets a public upstream (bjw-s-labs, controlplaneio-fluxcd). This only works if the jfroy/flatops/cluster GHCR package is public. Packages published with GITHUB_TOKEN inherit repo visibility, so it probably is, but worth verifying on the package settings page since a private package fails both the pull and the cosign signature lookup with a 401 that reads like a verification failure.

prune: Disabled needs the file removed in PR 4, not just left in place. The annotation stops kustomize-controller from deleting the object, but it does not stop it from applying it. Once FluxInstance switches to sync.kind: OCIRepository it will manage OCIRepository/flux-system too, and if this file is still in kubernetes/cluster/ both controllers will apply competing specs on every reconcile. Deleting the file in the final PR is what makes the annotation do its job — the object survives unowned and flux-operator adopts it.

Comment on lines +11 to +15

@claude claude Bot Aug 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No secretRef, so source-controller pulls anonymously. GHCR packages created by a GITHUB_TOKEN push default to private — unlike the other ghcr.io/jfroy/* images in this repo, which are all pulled without pull secrets because they've been flipped to public. If ghcr.io/jfroy/flatops/cluster was auto-created by the artifact-publisher run and nobody changed its visibility, this will sit in 401 Unauthorized rather than Ready.

Your rollout gate ("OCIRepository/flux-system is Ready and SourceVerified") catches it, but worth confirming the package visibility before merging rather than after.

verify:
provider: cosign
matchOIDCIdentity:
- issuer: ^https://token[.]actions[.]githubusercontent[.]com$
subject: ^https://github[.]com/jfroy/flatops/[.]github/workflows/cluster-release[.]yaml@refs/heads/main$
Comment on lines +15 to +20

@claude claude Bot Aug 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things to confirm before merging, both of which fail as "not Ready" on the root source rather than anywhere diagnosable:

  1. No secretRef. There is no image pull secret on any OCIRepository in this repo, so this only works if ghcr.io/jfroy/flatops/cluster is public. GHCR packages published by GITHUB_TOKEN are created private by default and do not inherit repository visibility — the package's visibility has to be flipped manually once. Worth adding to the rollout gate.

  2. --new-bundle-format=true in the sign step (line 84). That stores the signature as a sigstore bundle rather than the classic sha256-<digest>.sig tag. Please confirm the deployed source-controller's cosign verifier resolves it; cosign verify in the same job passing does not prove Flux can, since it uses a pinned cosign library version. If SourceVerified doesn't go true, dropping --new-bundle-format from both cosign sign and cosign verify is the fallback.

The identity regexes themselves match what the job asserts via --certificate-identity "${GITHUB_SERVER_URL}/${GITHUB_WORKFLOW_REF}".

Comment on lines +15 to +20

@claude claude Bot Aug 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No secretRef/serviceAccountName, so this relies on ghcr.io/jfroy/flatops/cluster being an anonymously pullable package — same as ghcr.io/jfroy/charts/zfs-static-csi. GHCR packages published via GITHUB_TOKEN are private on first publish regardless of repository visibility; visibility has to be flipped manually once. Worth adding to the rollout gate explicitly, since the symptom is OCIRepository stuck on a 401 with a UNAUTHORIZED message rather than an obvious auth-config error.

Loading