feat(ha): mirror worker credentials to Standby + prune reverse-diff (Fixes #295) - #2
Open
sumanthd032 wants to merge 3 commits into
Open
feat(ha): mirror worker credentials to Standby + prune reverse-diff (Fixes #295)#2sumanthd032 wants to merge 3 commits into
sumanthd032 wants to merge 3 commits into
Conversation
…Account, Role, RoleBinding)
Mirror the credential objects a promoted Standby needs to serve its
worker clusters without manual re-provisioning: worker-identity RBAC
(Role/RoleBinding/ServiceAccount — the only RBAC kinds
access_control_service.go ever creates; no ClusterRole or
ClusterRoleBinding exists to mirror, despite ADR Decision 6's broader
wording) and Secrets such as the gateway certificates the ovpn job
generates.
kubernetes.io/service-account-token Secrets are excluded: SA tokens are
signed by the issuing cluster's service-account key, so an Active-minted
token is cryptographically invalid on the Standby. Mirroring the
ServiceAccount is what matters — the Standby's own token controller
mints a locally-valid token for it.
Unlike the CRD set, these are core types that exist cluster-wide, so
every row is scoped in two layers:
- Server-side, the remote informers are scoped in cache.Options:
ServiceAccount/Role/RoleBinding by the same
util.LabelsKubeSliceController selector the Namespace informer already
uses (util.GetOwnerLabel embeds that exact key/value pair on every
credential object the controller creates), and Secret — which cannot
be label-scoped, the cert-generator job creates its Secrets unlabeled
— by a field selector excluding the SA-token type.
- Client-side, every row sets RequireMirroredNamespace: the object only
mirrors if its namespace is in the remote cache's label-scoped
Namespace view, i.e. a namespace the syncer itself mirrors. The
boundary is deliberately NOT name-based: under the Helm chart's
real-world --project-namespace-prefix ("kubeslice-"), the
controller's own kubeslice-controller namespace matches the
project-namespace naming pattern, and a prefix rule would have
mirrored its webhook TLS key, image-pull credentials, and Helm
release Secrets onto the Standby — found by running the Standby
against a Helm-installed Active hub. The label boundary is the one
ReconcileProjectNamespace actually maintains. A transient failure
reading the namespace surfaces as an error (workqueue retry), never
as a silent skip.
StripOwnerRefs is set on every row: ownerReferences resolve by UID,
which never survives a cross-cluster copy, and credential objects are
also written by actors outside this repo (token controller, cert job),
so the CRD set's audited only-VpnKeyRotation-needs-it reasoning cannot
hold here.
Verified live against the real two-hub Kind topology: worker SAs,
Roles, RoleBindings, and dashboard Secrets in both project namespaces
mirror with the sync label; SA-token Secrets and everything in
kubeslice-controller and kube-system stay off the Standby.
Part of kubeslice#295
Signed-off-by: Sumanth D <sumanthd032@gmail.com>
The prune loop so far only walked one direction — Standby mirrors whose Active-side original disappeared. Walk the other direction too: an Active-side object with no Standby mirror gets re-enqueued onto the ordinary workqueue, where the worker re-reads Active and runs the full Skip/namespace/conflict-guard chain. Three real cases produce that state, none of which the forward pass or the informers handle promptly: - a mirror someone deleted directly on the Standby (previously healed only by the informer's resync period, default 10 minutes); - an object whose RequireMirroredNamespace verdict was decided before its namespace's informer had delivered on cold start — a skip is terminal for that queue item, so without this it also waited for resync; - a key stuck deep in rate-limiter backoff after repeated failures (observed live: a fixed failure cause still took minutes to heal because the next retry was scheduled ~164s out; enqueue bypasses the rate limiter's delay, so recovery lands within one prune tick). Re-enqueueing is always safe: objects that should not mirror simply no-op through the guards again. Verified live: a mirrored Secret deleted directly on the Standby was re-created 15s later, within one --ha-sync-interval, with zero errors logged. Part of kubeslice#295 Signed-off-by: Sumanth D <sumanthd032@gmail.com>
…C sample The Standby now mirrors ha.FullMirrorSet — CRDMirrorSet plus CredentialMirrorSet. No new configuration: the mirrored-namespace gate derives entirely from the label boundary the controller already maintains, so the syncer needs no knowledge of the deployment's --project-namespace-prefix. config/ha/active-cluster-clusterrole.yaml gains read access to secrets/serviceaccounts (core) and roles/rolebindings (rbac), and the README's forward-looking credential-mirroring note becomes present-tense documentation. The Secret grant's security tradeoff is stated where it will be read rather than buried: RBAC cannot scope Secrets by .type or namespace label, and a ClusterRole binding is cluster-wide, so the Standby's identity can read every Secret on the Active hub — the syncer's field selector and mirrored-namespace gate narrow what gets copied, not what the identity could read. The narrower per-namespace RoleBinding alternative is documented alongside, with its maintenance cost. Grant coverage verified live via an impersonation can-i matrix: get/list/watch allowed for every mirrored type plus the HA Lease, writes and unrelated resources denied. Fixes kubeslice#295 Signed-off-by: Sumanth D <sumanthd032@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes kubeslice#295 — PR3 (final), stacked on PR2 (
feat/295-prune-sync-events). Mirrors the credential objects a promoted Standby needs to serve its workers without manual re-provisioning:CredentialMirrorSet— Secret / ServiceAccount / Role / RoleBinding. SA-token Secrets are excluded (tokens are signed per-cluster; the Standby's own controller mints a valid token for the mirrored SA). Two-layer scoping: informer-level label/field selectors, plus a client-sideRequireMirroredNamespacegate — an object mirrors only if its namespace is in the label-scoped Namespace view, not by name-prefix (the Helmkubeslice-prefix would otherwise match the controller's own namespace and mirror its TLS key / image-pull / Helm-release Secrets).StripOwnerRefson every row (UIDs never survive a cross-cluster copy; these objects are written by out-of-repo actors).--ha-sync-intervaltick instead of waiting on the 10-min resync.main.gomirrorsha.FullMirrorSet(no new config; the namespace gate derives from the existing label boundary).config/ha/active-cluster-clusterrole.yaml+ README gain secrets/serviceaccounts/roles/rolebindings reads, with the caveat stated inline: RBAC can't scope Secrets by.typeor namespace, so the identity can read every Active Secret — the field selector and namespace gate narrow what's copied, not what's readable.Base is
feat/295-prune-sync-events(PR2), notmaster, so the diff is exactly these 3 commits. Review after PR2.How Has This Been Tested?
go test -race ./pkg/ha/...(66 tests),go build ./...,go vet, andgofmt— all clean. Verified live against the two-hub Kind topology: credential mirror + SA-token filter + namespace scoping, prune reverse-diff re-create within ~15s, and an impersonationcan-iRBAC matrix all-green.Checklist:
Does this PR introduce a breaking change for other components like worker-operator?
No.