Repository navigation
hyperbytedb-operator v0.3.0 - #16
Merged
Merged
Conversation
Scale-down called /internal/drain and proceeded in the same reconcile pass: .spec.replicas dropped immediately and CleanupOrphanPVCs followed once the StatefulSet reported the lower count. That races region evacuation against volume deletion, and evacuation time is a function of how much data the node holds, so the existing 90/120s pod grace period is unrelated to it. Losing the race deletes the only remaining copy of whatever had not yet transferred. Scale-down now calls /internal/decommission -- drain means a restart and must not make a node ineligible to hold data -- and holds .spec.replicas until the departing node reports "leaving", which the node only reaches once no region still lists it. Nodes go one at a time, matching the order the StatefulSet removes them. The gate reads /health's "state" field, not "status": a node that is draining and one that has fully evacuated both report status "warn". An unreachable departing pod never gates. It may already be gone, and the cluster's own dead-peer path restores replication factor -- blocking forever on a pod that will never answer would wedge every later scale-down. Requires hyperbytedb with NodeState::Decommissioning and the /internal/decommission endpoint; deploy the server first.
AddPeer and RemovePeer are new ClusterRequest variants in the Raft log. A node on a build that predates one cannot decode it, rejects the whole append RPC, and stops replicating -- so a leader mid-rollout can lose control-plane quorum. Both ops ship behind opt-out flags for exactly this, and the documented procedure is to turn them off for the duration of a rolling upgrade of a sharded cluster. That procedure was impossible to follow through the operator. The CRD had no field for either flag, so the generated config never carried them and the server fell back to on, and reconcileConfigMap overwrites any hand edit on the next pass. The mitigation existed in the binary and not in the product. Both are *bool so unset stays absent rather than being written as an explicit true, which would take operator-side ownership of a server default. Tested in both directions: set values reach the config, unset values stay out of it. Found while building the mixed-version upgrade test -- testing whether the flag prevents the wedge is meaningless while the flag cannot be set.
The image bakes ENV HYPERBYTEDB__LOGGING__LEVEL=info, and the HYPERBYTEDB__* env overlay is applied OVER the mounted config.toml. The operator rendered the level into the ConfigMap and set no env var for it, so spec.logging.level was inert: setting it to debug changed the ConfigMap and the image default silently won. A sharded cluster could not be put into debug at all, which is exactly what was needed to diagnose a placement and staging failure on a live cluster. Set it as env, the way this block already restates the four other image defaults, so the CR field wins regardless of what the image bakes. Unset still pins info explicitly rather than letting the deployed image tag decide the operator's behaviour.
reconcileClusterMembership skipped any pod the leader already knew, so a node scaled down and back up -- still listed as `leaving` -- never got add-node and the database never re-admitted it. Departed states no longer count as known. Safe against undoing a scale-down: the views cover only ordinals below spec.replicas, so a pod still being evacuated is never considered.
Scale-down decommissioned departing nodes and told survivors they left, but never called remove-node. Shrinking 6 -> 3 left 3 of 6 voters alive: no quorum, nothing committed. The departing voters are now removed, highest ordinal first and re-finding the leader each time (removing the leader makes it step down), before .spec.replicas shrinks. A failed removal holds the scale-down.
Under write saturation /ping on the main port queued behind write work past the 3s liveness timeout, and kubelet killed busy but healthy nodes. A clustered node now gets control_port = server port + 1 in its config; the database serves /ping, /health and Raft RPCs there on a dedicated runtime. Liveness moves to that port (plain HTTP); readiness stays on the main port, since it should reflect serving capacity. Single-node clusters are unchanged.
380fdd3 moved liveness to the control port for every clustered CR. Only database builds with the control listener serve that port, so an existing cluster on 0.8.5-beta failed liveness on a port nothing listened on and restarted in a loop for three days. The control port (config, container port and liveness) is now used only when the CR has hyperbytedb.hyperbyte.cloud/control-listener: "true".
A 12 -> 3 scale-down on n3 evacuated the departing nodes and removed them from the Raft voters, then stalled with 12 pods. Removed nodes keep campaigning and report unready, and with OrderedReady the StatefulSet controller will not delete a pod while a lower-ordinal pod is unready, so pods 3..11 stayed forever. .spec.replicas only shrinks after evacuation and voter removal, so every pod at or above it is already out of the cluster. The operator now deletes the unready ones itself; ready ones are left to the StatefulSet.
A clustered StatefulSet is OrderedReady, and a departing node reports unready, so the StatefulSet will not create a pod above one: on n3 a pod deleted during a scale-down was never recreated, and a full restart of 8 pods took over an hour, one at a time. A cluster annotated hyperbytedb.hyperbyte.cloud/parallel-pods: "true" gets Parallel. Opt-in, because the policy cannot change in place: the controller deletes the StatefulSet with its pods orphaned (left running) and recreates it, and existing clusters must not see that unasked.
A scale-down decommissions departing nodes one at a time, so a node that has not been told yet is still a placement target: on n3 an 8 -> 3 scale-down moved node 8's copies onto nodes 4-7, all leaving too, and node 6 then held 61 copies to move. With the annotation hyperbytedb.hyperbyte.cloud/evacuate-together: "true" every departing node is told at once, so each copy moves once, onto a node that stays. spec.replicas still shrinks only after all have evacuated. Opt-in, because an older database may not expect several nodes decommissioning at once. The decision is departurePlan, tested for both modes.
With no pool field set the config said pool_size = 1, one chDB connection for both queries and writes. A sharded query runs one chDB query per region it touches, so a few concurrent multi-region queries queued: on n3, 4 concurrent 10-host GROUP BY queries ran at 250-450 ms p50 with one query connection and 90-130 ms with 4. Unset pools now get a query pool of 12 and keep the write pool at 1. A spec that sets poolSize, queryPoolSize or writePoolSize renders exactly as before. The webhook no longer stamps poolSize: 1, which would have kept every new cluster at one connection.
One operator manages clusters on several HyperbyteDB versions, and the config keys they accept differ in both directions: 0.9 rejects keys it does not know, while 0.8 ignores them and runs on its defaults. 25812f2 wrote query_pool_size = 12 for every cluster without a pool field; a 0.8 cluster would have ignored it and run on pool_size's default instead. The operator now resolves the version a cluster runs (the database-version annotation, spec.version, or the image tag, the first that parses; else the newest it knows) and renders only what that version supports, from CompatibilityMatrix in compat.go: split chDB pools, the database-sized query pool, the [sharding] section, the control listener and evacuate-together are all 0.9.0-beta and later. A 0.8 cluster gets pool_size and none of the rest. The DatabaseVersion status condition says which version was used, where it came from and what was left out. For 0.9 with no pool field the query pool is now left unset, so the database sizes it from its CPU limit (and SET query_pool_size changes it); this replaces 25812f2's fixed 12, which on 3-CPU pods was 10-40% slower than 4. Rendered config for both live n3 clusters is unchanged (0.8.5-beta with poolSize 6, and the e2e cluster), so upgrading the operator does not restart them. docs/compatibility.md documents the matrix; the chart is 0.2.0.
With spec.auth.enabled on HyperbyteDB 0.9.0-beta+ (capability cluster-auth), the operator keeps a Secret (<cluster>-auth, or spec.auth.credentialsSecretName) with cluster-token, admin-username and admin-password, generating missing keys and keeping existing ones. Pods get the keys as HYPERBYTEDB__AUTH__* env, the drain hook sends the token, and the operator's own calls send it via a context-carried token on a shared client. Clusters with auth off, or on older versions, render exactly as before (config and StatefulSet byte-identical for both n3 clusters).
… timeout Both defaulted to 1000, which Raft refuses: a cluster created without them crash-looped on every node (found bringing up UAT). The heartbeat now defaults to 300, as both n3 clusters already set, and a CEL rule rejects a heartbeat at or above the election timeout. The chart's CRD copy, stale since before v0.2.0, is synced from config/crd/bases.
0.9.0-beta has no auth.cluster_token and refuses unknown config keys, so the auth env would stop its pods starting. Newest known database is now 0.9.1-beta; chart 0.3.0.
austin-barrington
force-pushed
the
release/v0.3.0
branch
from
October 13, 2026 21:45
99e9607 to
ef04689
Compare
golangci-lint (v2.8.0, built with Go 1.25) failed the PR on nine issues: unchecked fmt.Fprintf / Body.Close errors in tests, repeated literals (container name, http port name, default log level, annotation true) now constants, and sort.Slice replaced by slices.Sort.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Operator for HyperbyteDB 0.9.1-beta.
spec.versionor image tag). Seedocs/compatibility.md.spec.auth.enabledcreates an auth Secret (cluster token and bootstrap admin) and passes it to the pods. Needs HyperbyteDB 0.9.1-beta.spec.logging.levelreaches the process;Upgrade note: a cluster with auth on that runs 0.9.0-beta must move to 0.9.1-beta (version and image) before or together with this operator. Otherwise its pods lose the auth env.
Tests:
go test ./internal/...andgo vetpass.make lintdidn't run locally (the linter was built with Go 1.24, the repo targets 1.25.3).