Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
8fdc199
Wait for evacuation before shrinking a sharded cluster
austin-barrington Sep 9, 2026
f2482d8
Expose the sharding upgrade gates on the CRD
austin-barrington Sep 10, 2026
763bbab
Make spec.logging.level actually reach the process
austin-barrington Sep 12, 2026
8a2b103
controller: send add-node for pods the leader lists as departed
austin-barrington Sep 22, 2026
5e45a49
controller: remove departing nodes from the Raft voter set on scale-down
austin-barrington Sep 24, 2026
efb900d
statefulset: probe liveness on the database's control listener
austin-barrington Sep 24, 2026
be09412
statefulset: make the control-port liveness probe opt-in
austin-barrington Sep 27, 2026
12fa019
controller: delete unready pods a scale-down leaves behind
austin-barrington Sep 28, 2026
226fdd9
statefulset: opt-in Parallel pod management for clusters
austin-barrington Oct 9, 2026
c76cfed
controller: opt-in evacuation of all departing nodes together
austin-barrington Oct 10, 2026
47ed693
hyperbytedb: default the chDB query pool to 12 connections
austin-barrington Oct 10, 2026
c7faf77
hyperbytedb: render each cluster for the HyperbyteDB version it runs
austin-barrington Oct 10, 2026
8e5cc5c
Cluster auth: Secret, pod env and the cluster token
austin-barrington Oct 11, 2026
397355d
CRD: Raft heartbeat defaults to 300 ms and must be below the election…
austin-barrington Oct 11, 2026
ef04689
v0.3.0: cluster auth needs HyperbyteDB 0.9.1-beta
austin-barrington Oct 13, 2026
e12dc9f
Lint: checked errors, named constants, slices.Sort
austin-barrington Oct 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ configurable values.
```sh
helm upgrade hyperbytedb-operator \
oci://ghcr.io/hyperbyte-cloud/charts/hyperbytedb-operator \
--version 0.2.0 \
--version 0.3.0 \
--namespace hyperbytedb-operator-system \
--reuse-values
```
Expand Down Expand Up @@ -142,6 +142,13 @@ The operator manages three CRDs:

See `config/samples/` for example manifests.

## HyperbyteDB versions

The operator works out which HyperbyteDB version each cluster runs (from
`spec.version`, the image tag, or the `hyperbytedb.hyperbyte.cloud/database-version`
annotation) and renders only what that version supports. See
[docs/compatibility.md](docs/compatibility.md) for the matrix.

## Development

```sh
Expand Down
26 changes: 24 additions & 2 deletions api/v1alpha1/hyperbytedbcluster_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -230,7 +230,8 @@ type ChDBSpec struct {
SessionDataPath string `json:"sessionDataPath,omitempty"`

// Sets both query and write pool sizes when QueryPoolSize and WritePoolSize
// are unset. Defaults to 1 when all pool fields are unset.
// are unset. When all pool fields are unset the query pool is 12 and the
// write pool 1.
// +optional
// +kubebuilder:validation:Minimum=1
PoolSize int32 `json:"poolSize,omitempty"`
Expand Down Expand Up @@ -265,6 +266,7 @@ type LoggingSpec struct {
Format string `json:"format,omitempty"`
}

// +kubebuilder:validation:XValidation:rule="!has(self.raftHeartbeatIntervalMs) || !has(self.raftElectionTimeoutMs) || self.raftHeartbeatIntervalMs < self.raftElectionTimeoutMs",message="raftHeartbeatIntervalMs must be below raftElectionTimeoutMs"
type ClusterTuningSpec struct {
// +kubebuilder:default=2
HeartbeatIntervalSecs int32 `json:"heartbeatIntervalSecs,omitempty"`
Expand All @@ -275,7 +277,10 @@ type ClusterTuningSpec struct {
// +kubebuilder:default=5
ReplicationMaxRetries int32 `json:"replicationMaxRetries,omitempty"`

// +kubebuilder:default=1000
// Must be below raftElectionTimeoutMs: Raft refuses to start otherwise.
// The default was 1000, equal to the election timeout's, and a cluster
// that kept both defaults crash-looped.
// +kubebuilder:default=300
RaftHeartbeatIntervalMs int32 `json:"raftHeartbeatIntervalMs,omitempty"`

// +kubebuilder:default=1000
Expand Down Expand Up @@ -388,6 +393,23 @@ type ShardingSpec struct {
// +kubebuilder:validation:Minimum=1
ScatterMaxPeerAttempts int32 `json:"scatterMaxPeerAttempts,omitempty"`

// Propose AddPeer shard-map ops. Defaults to true when unset.
//
// Set false for the duration of a rolling upgrade of a sharded cluster: a
// node running a build that predates the op cannot decode it, rejects the
// Raft append RPC carrying it, and stops replicating until every node is
// upgraded. Re-enable once the rollout completes.
// +optional
AddPeerProposalsEnabled *bool `json:"addPeerProposalsEnabled,omitempty"`

// Propose RemovePeer shard-map ops. Defaults to true when unset.
//
// Carries the identical rolling-upgrade hazard as AddPeerProposalsEnabled,
// and scale-down evacuation depends on it: leaving it false prevents a
// decommissioning node from ever releasing its regions.
// +optional
RemovePeerProposalsEnabled *bool `json:"removePeerProposalsEnabled,omitempty"`

// Load-based split QPS threshold; 0 = disabled. Emitted only when set so the
// server default (0) applies when omitted.
// +optional
Expand Down
4 changes: 0 additions & 4 deletions api/v1alpha1/hyperbytedbcluster_webhook.go
Original file line number Diff line number Diff line change
Expand Up @@ -68,10 +68,6 @@ func (w *HyperbytedbClusterWebhook) Default(_ context.Context, obj *HyperbytedbC
}
}

if obj.Spec.ChDB.PoolSize == 0 {
obj.Spec.ChDB.PoolSize = 1
}

if obj.Spec.Retention.Interval == "" {
obj.Spec.Retention.Interval = "60s"
}
Expand Down
10 changes: 10 additions & 0 deletions api/v1alpha1/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
Expand Up @@ -2979,7 +2979,8 @@ spec:
poolSize:
description: |-
Sets both query and write pool sizes when QueryPoolSize and WritePoolSize
are unset. Defaults to 1 when all pool fields are unset.
are unset. When all pool fields are unset the query pool is 12 and the
write pool 1.
format: int32
minimum: 1
type: integer
Expand Down Expand Up @@ -3023,7 +3024,11 @@ spec:
format: int32
type: integer
raftHeartbeatIntervalMs:
default: 1000
default: 300
description: |-
Must be below raftElectionTimeoutMs: Raft refuses to start otherwise.
The default was 1000, equal to the election timeout's, and a cluster
that kept both defaults crash-looped.
format: int32
type: integer
raftSnapshotThreshold:
Expand Down Expand Up @@ -3131,6 +3136,10 @@ spec:
- enabled
type: object
type: object
x-kubernetes-validations:
- message: raftHeartbeatIntervalMs must be below raftElectionTimeoutMs
rule: '!has(self.raftHeartbeatIntervalMs) || !has(self.raftElectionTimeoutMs)
|| self.raftHeartbeatIntervalMs < self.raftElectionTimeoutMs'
failover:
description: FailoverSpec controls automatic failure detection and
recovery.
Expand Down Expand Up @@ -3595,6 +3604,15 @@ spec:
block into config.toml. A 1-replica CR with sharding.enabled is a
1-member cluster ([cluster] enabled=true). Defaults stay off.
properties:
addPeerProposalsEnabled:
description: |-
Propose AddPeer shard-map ops. Defaults to true when unset.

Set false for the duration of a rolling upgrade of a sharded cluster: a
node running a build that predates the op cannot decode it, rejects the
Raft append RPC carrying it, and stops replicating until every node is
upgraded. Re-enable once the rollout completes.
type: boolean
bootstrapTimeoutMs:
description: Sync bootstrap RPC timeout.
format: int64
Expand Down Expand Up @@ -3644,6 +3662,14 @@ spec:
format: int64
minimum: 1
type: integer
removePeerProposalsEnabled:
description: |-
Propose RemovePeer shard-map ops. Defaults to true when unset.

Carries the identical rolling-upgrade hazard as AddPeerProposalsEnabled,
and scale-down evacuation depends on it: leaving it false prevents a
decommissioning node from ever releasing its regions.
type: boolean
replicationFactor:
description: Target replica count per shard region.
format: int32
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ spec:
maxPointsPerBatch: 50000
chdb:
sessionDataPath: /var/lib/hyperbytedb/chdb
poolSize: 1
queryPoolSize: 12
retention:
enabled: true
interval: 60s
Expand Down
4 changes: 2 additions & 2 deletions dist/chart/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ type: application
# release workflow (.github/workflows/release.yml) stamps them with the git
# tag (or a `0.0.0-sha.<short>` snapshot for main branch builds) when the
# chart is packaged and pushed to ghcr.io.
version: 0.1.2
appVersion: "0.1.2"
version: 0.3.0
appVersion: "0.3.0"

home: https://hyperbyte-cloud.github.io/hyperbytedb/
sources:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2979,7 +2979,8 @@ spec:
poolSize:
description: |-
Sets both query and write pool sizes when QueryPoolSize and WritePoolSize
are unset. Defaults to 1 when all pool fields are unset.
are unset. When all pool fields are unset the query pool is 12 and the
write pool 1.
format: int32
minimum: 1
type: integer
Expand Down Expand Up @@ -3023,7 +3024,11 @@ spec:
format: int32
type: integer
raftHeartbeatIntervalMs:
default: 1000
default: 300
description: |-
Must be below raftElectionTimeoutMs: Raft refuses to start otherwise.
The default was 1000, equal to the election timeout's, and a cluster
that kept both defaults crash-looped.
format: int32
type: integer
raftSnapshotThreshold:
Expand Down Expand Up @@ -3131,6 +3136,10 @@ spec:
- enabled
type: object
type: object
x-kubernetes-validations:
- message: raftHeartbeatIntervalMs must be below raftElectionTimeoutMs
rule: '!has(self.raftHeartbeatIntervalMs) || !has(self.raftElectionTimeoutMs)
|| self.raftHeartbeatIntervalMs < self.raftElectionTimeoutMs'
failover:
description: FailoverSpec controls automatic failure detection and
recovery.
Expand Down Expand Up @@ -3592,8 +3601,18 @@ spec:
sharding:
description: |-
Experimental series sharding. When set, the operator writes a `[sharding]`
block into config.toml. Enabling it requires replicas > 1 (cluster mode).
block into config.toml. A 1-replica CR with sharding.enabled is a
1-member cluster ([cluster] enabled=true). Defaults stay off.
properties:
addPeerProposalsEnabled:
description: |-
Propose AddPeer shard-map ops. Defaults to true when unset.

Set false for the duration of a rolling upgrade of a sharded cluster: a
node running a build that predates the op cannot decode it, rejects the
Raft append RPC carrying it, and stops replicating until every node is
upgraded. Re-enable once the rollout completes.
type: boolean
bootstrapTimeoutMs:
description: Sync bootstrap RPC timeout.
format: int64
Expand Down Expand Up @@ -3643,6 +3662,14 @@ spec:
format: int64
minimum: 1
type: integer
removePeerProposalsEnabled:
description: |-
Propose RemovePeer shard-map ops. Defaults to true when unset.

Carries the identical rolling-upgrade hazard as AddPeerProposalsEnabled,
and scale-down evacuation depends on it: leaving it false prevents a
decommissioning node from ever releasing its regions.
type: boolean
replicationFactor:
description: Target replica count per shard region.
format: int32
Expand Down
33 changes: 30 additions & 3 deletions dist/install.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3182,7 +3182,8 @@ spec:
poolSize:
description: |-
Sets both query and write pool sizes when QueryPoolSize and WritePoolSize
are unset. Defaults to 1 when all pool fields are unset.
are unset. When all pool fields are unset the query pool is 12 and the
write pool 1.
format: int32
minimum: 1
type: integer
Expand Down Expand Up @@ -3226,7 +3227,11 @@ spec:
format: int32
type: integer
raftHeartbeatIntervalMs:
default: 1000
default: 300
description: |-
Must be below raftElectionTimeoutMs: Raft refuses to start otherwise.
The default was 1000, equal to the election timeout's, and a cluster
that kept both defaults crash-looped.
format: int32
type: integer
raftSnapshotThreshold:
Expand Down Expand Up @@ -3334,6 +3339,10 @@ spec:
- enabled
type: object
type: object
x-kubernetes-validations:
- message: raftHeartbeatIntervalMs must be below raftElectionTimeoutMs
rule: '!has(self.raftHeartbeatIntervalMs) || !has(self.raftElectionTimeoutMs)
|| self.raftHeartbeatIntervalMs < self.raftElectionTimeoutMs'
failover:
description: FailoverSpec controls automatic failure detection and
recovery.
Expand Down Expand Up @@ -3795,8 +3804,18 @@ spec:
sharding:
description: |-
Experimental series sharding. When set, the operator writes a `[sharding]`
block into config.toml. Enabling it requires replicas > 1 (cluster mode).
block into config.toml. A 1-replica CR with sharding.enabled is a
1-member cluster ([cluster] enabled=true). Defaults stay off.
properties:
addPeerProposalsEnabled:
description: |-
Propose AddPeer shard-map ops. Defaults to true when unset.

Set false for the duration of a rolling upgrade of a sharded cluster: a
node running a build that predates the op cannot decode it, rejects the
Raft append RPC carrying it, and stops replicating until every node is
upgraded. Re-enable once the rollout completes.
type: boolean
bootstrapTimeoutMs:
description: Sync bootstrap RPC timeout.
format: int64
Expand Down Expand Up @@ -3846,6 +3865,14 @@ spec:
format: int64
minimum: 1
type: integer
removePeerProposalsEnabled:
description: |-
Propose RemovePeer shard-map ops. Defaults to true when unset.

Carries the identical rolling-upgrade hazard as AddPeerProposalsEnabled,
and scale-down evacuation depends on it: leaving it false prevents a
decommissioning node from ever releasing its regions.
type: boolean
replicationFactor:
description: Target replica count per shard region.
format: int32
Expand Down
70 changes: 70 additions & 0 deletions docs/compatibility.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# HyperbyteDB compatibility

One operator manages clusters on several HyperbyteDB versions. For each
cluster it works out which version runs and renders the config and pods for
that version: a HyperbyteDB 0.9 rejects config keys it does not know, and a
0.8 ignores them and runs on its defaults, so neither is safe to get keys
meant for the other.

## Which version a cluster runs

The first of these that is a version (`0.8.5-beta`, `v0.9.0-beta`, `0.9.0`):

1. the annotation `hyperbytedb.hyperbyte.cloud/database-version`,
2. `spec.version`,
3. the tag of the database image.

If none is (a local image tagged `e2e`, say), the cluster is rendered for the
newest version this operator knows. The `DatabaseVersion` condition in the
cluster's status says which version was used, where it came from, and what
was left out for it:

```sh
kubectl get hyperbytedbcluster my-cluster \
-o jsonpath='{.status.conditions[?(@.type=="DatabaseVersion")]}'
```

Reasons: `Supported`, `Assumed` (no version found), `NewerThanKnown`
(rendered for the newest this operator knows), `OlderThanSupported`
(condition status `False`; rendered as the oldest supported).

## Matrix

| Operator | Renders for HyperbyteDB | Newest known |
|---|---|---|
| v0.3.0 | 0.8.0 and later | 0.9.1-beta |
| v0.2.0 | 0.8.0 and later | 0.9.0-beta |
| v0.1.x | not version-aware: renders the newest keys for every cluster (0.8.x ignores the ones it does not know) | — |

What v0.3.0 renders per version:

| Capability | HyperbyteDB | For older versions the operator |
|---|---|---|
| Split chDB pools (`query_pool_size`, `write_pool_size`) | 0.9.0-beta | writes `pool_size` for both: `spec.chdb.poolSize`, else `queryPoolSize`, else 1 |
| Query pool sized by the database from its CPUs | 0.9.0-beta | writes `pool_size = 1` when no pool field is set, as v0.1.x did |
| Sharding (`[sharding]`, `spec.sharding`) | 0.9.0-beta | omits the section; `spec.sharding` has no effect |
| Control listener (`control-listener` annotation) | 0.9.0-beta | ignores the annotation; liveness stays on the server port |
| Evacuate together (`evacuate-together` annotation) | 0.9.0-beta | ignores the annotation; departing nodes leave one at a time |
| Cluster auth (`spec.auth.enabled`: auth Secret, cluster token, bootstrap admin) | 0.9.1-beta | renders `auth.enabled` only; no Secret, so a cluster with auth on cannot reach its own internal routes |
| Parallel pod management (`parallel-pods` annotation) | any | — (operator only) |

The source of truth is `CompatibilityMatrix` in
`internal/hyperbytedb/compat.go`; keep this page in step with it.

## Upgrading

- v0.2.0 → v0.3.0: cluster auth needs HyperbyteDB 0.9.1-beta (0.9.0-beta
has no cluster token and refuses the key). A cluster with
`spec.auth.enabled` that resolves to 0.9.0-beta gets no auth Secret env,
so move it to 0.9.1-beta (version and image) before or with the operator.
- Upgrading the operator does not restart a cluster unless its rendered
config or pods change. v0.1.x → v0.2.0 renders the same config for a
0.8.5-beta cluster and for a 0.9 development cluster that sets its pool
sizes; a 0.9 cluster with no pool field changes (it stops writing
`pool_size = 1` and lets the database size its query pool), which rolls
its pods once.
- Upgrading HyperbyteDB: change `spec.version` (or the image). The operator
renders for the new version in the same reconcile, so the new pods start
with a config they accept.
- Do not run `SET` statements while a HyperbyteDB rolling upgrade is in
progress: nodes on the older build cannot decode them.
Loading
Loading