Skip to content

HyperbyteDB 0.9.1-beta #116

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
austin-barrington merged 233 commits into from
Oct 17, 2026
Merged

HyperbyteDB 0.9.1-beta #116

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
233 commits
Select commit Hold shift + click to select a range
f7b550a
Allow a one-member sharded cluster to own regions.
austin-barrington Sep 4, 2026
8bdbd99
Install the committed shard map on join before region movement.
austin-barrington Sep 4, 2026
3dc712c
Stage existing region data onto a live joiner (P1.3)
austin-barrington Sep 6, 2026
d13af99
Place region primaries on joined members (P1.4-P1.6)
austin-barrington Sep 6, 2026
84d7d50
Re-read the region before each placement transfer
austin-barrington Sep 6, 2026
4d9282c
Fix review blockers in region placement (#1-#7)
austin-barrington Sep 6, 2026
71e677f
Stop placement stalling on one lagging candidate (#9-#12)
austin-barrington Sep 6, 2026
1a1edf6
Say what idle-member placement actually triggers on (#8)
austin-barrington Sep 6, 2026
66cbd50
Default the AddPeer gate on
austin-barrington Sep 7, 2026
d6829da
Add a frozen hash for region placement
austin-barrington Sep 8, 2026
29c1ad4
Add single-step placement diff toward a target peer set
austin-barrington Sep 8, 2026
7a33548
Add rendezvous-hash target placement
austin-barrington Sep 8, 2026
382b167
Add RemovePeer shard-map op
austin-barrington Sep 8, 2026
0b283f2
Add RemovePeer gate and movement budget config
austin-barrington Sep 8, 2026
bac4aa4
Extract region staging and the caught-up probe from live placement
austin-barrington Sep 8, 2026
3ca4bdc
Add linear-scaling acceptance benchmark
austin-barrington Sep 8, 2026
e62bd19
Add Decommissioning state, membership predicates, guarded transitions
austin-barrington Sep 8, 2026
32403ba
Bring the design docs onto the implementation branch
austin-barrington Sep 8, 2026
dffbcb2
Classify region movements into priority tiers
austin-barrington Sep 8, 2026
1b31f48
Require sustained probe failure before demoting a node
austin-barrington Sep 8, 2026
8318016
Bootstrap regions through the shared placement function
austin-barrington Sep 8, 2026
e5ccfd5
Replace recency-keyed placement with convergence toward a target
austin-barrington Sep 8, 2026
74d76db
Drain hands off primaries; decommission evacuates and leaves
austin-barrington Sep 8, 2026
24c8789
Place the right split child instead of cloning the parent
austin-barrington Sep 8, 2026
05778e2
Budget convergence-tier region movement
austin-barrington Sep 9, 2026
32c627f
Retire a decommissioning node once its regions have actually moved
austin-barrington Sep 9, 2026
6264ca1
Remove dead peers so convergence can replace them
austin-barrington Sep 9, 2026
97e9543
Retire the superseded placement policy and its tests
austin-barrington Sep 9, 2026
8257036
Stop scale-down losing data, found by the kind e2e
austin-barrington Sep 10, 2026
f874130
Stop the scale-down e2e reporting a false pass
austin-barrington Sep 10, 2026
039584e
Add a full sharding lifecycle e2e with query-correctness gates
austin-barrington Sep 10, 2026
562735f
Plan five real-world sharding test gaps
austin-barrington Sep 10, 2026
2fde78f
Assert exactly-once for writes during a live scale-down
austin-barrington Sep 10, 2026
a07d7eb
Extend write-churn to three topology changes
austin-barrington Sep 10, 2026
475ac53
Stop the leader re-sync loop firing on signals sharding invalidates
austin-barrington Sep 11, 2026
b57c1d3
Judge a peer's map probe against the version it was probed against
austin-barrington Sep 11, 2026
7aade1f
Stop an unreadable metric from passing as a zero one
austin-barrington Sep 11, 2026
5e20a2e
Never rebuild port-forwards from inside a command substitution
austin-barrington Sep 11, 2026
d8a9a23
Fix three harness defects that failed runs whose gates all passed
austin-barrington Sep 11, 2026
54a6153
Make a failing placement say why, instead of failing silently
austin-barrington Sep 12, 2026
77add80
Carry the rejection reason instead of a bare status code
austin-barrington Sep 13, 2026
7feaa3d
Stop D1 failing on a read taken before the cluster converged
austin-barrington Sep 13, 2026
7f70582
Stop a missing series row from silently corrupting transferred points
austin-barrington Sep 13, 2026
d84d33e
Prune per-region counters for regions the map no longer has
austin-barrington Sep 13, 2026
b38d912
Evict known_series when a series-id range is dropped
austin-barrington Sep 13, 2026
ad621cc
Poll in D5 too, not just D1
austin-barrington Sep 13, 2026
771ffe3
Never publish a split child whose primary holds no data
austin-barrington Sep 13, 2026
ab4c176
Give a region a serving state, and route only on serving regions
austin-barrington Sep 13, 2026
720e3b9
Publish a split child staging, and cut it over atomically
austin-barrington Sep 13, 2026
7235866
Scatter over the serving view, not a raw map snapshot
austin-barrington Sep 13, 2026
a639796
Never drop the donor's rows while the child is still staging
austin-barrington Sep 14, 2026
bb2233f
Resolve a staging region by identity, not by extending the donor's range
austin-barrington Sep 14, 2026
b4d4567
Add a suite that splits while writes are in flight
austin-barrington Sep 14, 2026
b2fadc9
Fix the two causes of staging failure under split-under-write load
austin-barrington Sep 14, 2026
565e27b
Never move a split child using its provisional id
austin-barrington Sep 14, 2026
ff909d9
Make a split metadata-only, TiKV-style
austin-barrington Sep 14, 2026
a4ba0a1
Commit a new replica as a learner before copying to it
austin-barrington Sep 14, 2026
51c170d
Keep a learner out of the primary role, and finish abandoned learners
austin-barrington Sep 15, 2026
2a1d9e4
Apply the failover watermark guard to planned promotion too
austin-barrington Sep 15, 2026
2cec64b
Allow bounding cargo parallelism in the image build
austin-barrington Sep 15, 2026
39ae3e6
Do not let the promotion watermark guard block evacuation
austin-barrington Sep 15, 2026
29c3c5b
Stop the learner sweep re-staging mid-flight learners every tick
austin-barrington Sep 15, 2026
0dc8882
sharding: make convergence run, spread primaries, stop partial writes…
austin-barrington Sep 20, 2026
b18d4fe
e2e: fix checks that passed without testing what they claimed
austin-barrington Sep 20, 2026
0f641de
e2e: clear the dataset between linearity sweep points
austin-barrington Sep 20, 2026
276329a
sharding: bound the mover pool by bytes in flight, not job count
austin-barrington Sep 22, 2026
e423e45
cluster: re-admit a departed node when the operator re-adds it
austin-barrington Sep 22, 2026
3b3dfd3
cluster: let departed and restarted nodes rejoin cleanly
austin-barrington Sep 24, 2026
6f98882
sharding: remove a departing non-primary before moving primacy
austin-barrington Sep 24, 2026
eb339e7
sharding: drop shard spaces when a clustered drop is applied
austin-barrington Sep 24, 2026
b45a08f
linearity: benchmark from inside the cluster
austin-barrington Sep 24, 2026
35fdd01
sharding: split on the median a region peer reports
austin-barrington Sep 24, 2026
b874c4e
linearity: wait for the space to split before measuring
austin-barrington Sep 24, 2026
07fd168
linearity: saturate with several clients, fresh database per point
austin-barrington Sep 24, 2026
a5d3fd5
cluster: stay up and keep regions moving under write saturation
austin-barrington Sep 24, 2026
9f17525
linearity: retry shed writes, report why writes failed, check the layout
austin-barrington Sep 24, 2026
1de1598
hinted_handoff: count queued hints in memory, not by scanning
austin-barrington Sep 24, 2026
890f135
chdb: probe a table's engine columns once, not on every write
austin-barrington Sep 24, 2026
81f8709
Add an optional in-process CPU profiler
austin-barrington Sep 24, 2026
0c3b17a
sharding: forward a write's regions in one request per primary node
austin-barrington Sep 24, 2026
f996c6b
Cargo.lock: keep chdb-rust at the pinned 1.4.0
austin-barrington Sep 24, 2026
09408cc
chdb: probe until the schema cache is verified, not until materialized
austin-barrington Sep 27, 2026
0fdbef2
tests: stop sharding tests depending on replication being slow
austin-barrington Sep 27, 2026
7e8bb92
sharding: balance primaries by load across nodes
austin-barrington Sep 27, 2026
04846e8
sharding: balance region copies across nodes, not just primaries
austin-barrington Sep 27, 2026
1956835
cluster: forward and replicate points as binary frames
austin-barrington Sep 27, 2026
1647623
linearity: wait for the whole layout to settle before measuring
austin-barrington Sep 27, 2026
6a6bc28
sharding: create new measurements already split across the nodes
austin-barrington Sep 27, 2026
38e28ca
linearity: allow up to 40 minutes for pods to reach N
austin-barrington Sep 27, 2026
0f056c5
sharding: stop balance from piling learners onto one node
austin-barrington Sep 28, 2026
1bb9e7e
replication: send each peer its queued parts in one request
austin-barrington Sep 28, 2026
f1e4d3c
chdb: add a writer's columns even when another writer created the table
austin-barrington Sep 28, 2026
7ab3dfc
metadata: seek series range scans to the start of the range
austin-barrington Sep 28, 2026
4a4525e
linearity: make the client batch size a sweep setting
austin-barrington Sep 28, 2026
a672b35
sharding: append a forwarded batch's parts as one WAL entry
austin-barrington Sep 28, 2026
8de48ab
config: admit 16 client writes per CPU by default, not 8
austin-barrington Sep 28, 2026
fcd8c03
metrics: time each stage of the sharded write path
austin-barrington Sep 28, 2026
54d3cb6
sharding: append the local share of a write while forwards are in flight
austin-barrington Sep 28, 2026
b92cda5
metrics: time the WAL append's sub-stages and the runtime's wake delay
austin-barrington Sep 29, 2026
d4f2d30
sharding: acknowledge writes once their forwards are queued
austin-barrington Sep 29, 2026
02c5303
cluster: a departing node retires only after its forward queue drains
austin-barrington Sep 29, 2026
5cfa5d6
sharding: drain the forward queue in small batches
austin-barrington Sep 29, 2026
14eb4bf
config: turn async forwarding off by default until it beats synchronous
austin-barrington Sep 29, 2026
3efbedd
telemetry: optional OTLP request tracing for the write path
austin-barrington Sep 29, 2026
1235c39
telemetry: record the WAL batcher's queue and write time on traced ap…
austin-barrington Sep 29, 2026
8ffdee9
linearity: retry writes the server refused before reading their body
austin-barrington Sep 30, 2026
164220b
query: read results as streamed Arrow batches, not JSONEachRow text
austin-barrington Sep 30, 2026
2fe0872
Retry only the regions a primary refused at capacity
austin-barrington Sep 30, 2026
71ca6f7
Fix multi-region GROUP BY time, partial name clashes, series order
austin-barrington Sep 30, 2026
0203953
Strict time bounds exclude their instant
austin-barrington Sep 30, 2026
e7f7f13
SHOW FIELD KEYS gathers fields from every region
austin-barrington Sep 30, 2026
c53fafe
Inferred column mapping never takes time for a tag
austin-barrington Sep 30, 2026
13e12fc
Query every region when the tag keys are unknown
austin-barrington Sep 30, 2026
0d337d2
Show the cause of failed forwards
austin-barrington Sep 30, 2026
3425575
Read a refused write's body before sending the 503
austin-barrington Sep 30, 2026
271a586
InfluxQL column names and time column
austin-barrington Oct 1, 2026
7368664
Aggregates without GROUP BY time() are stamped epoch 0
austin-barrington Oct 1, 2026
b9c9e10
Rust ingest bench for the linearity sweep
austin-barrington Oct 1, 2026
096021d
CLI e2e: name the materialized view column
austin-barrington Oct 1, 2026
f2202c6
Sweep: the CLI lives at /usr/local/bin in the runtime image
austin-barrington Oct 1, 2026
be3c631
Share one forward request per node among concurrent writes
austin-barrington Oct 1, 2026
7c73399
Make forward coalescing opt-in (sharding.forward_coalesce)
austin-barrington Oct 1, 2026
010acbb
Optional HTTP/2 between nodes (cluster.internal_http2)
austin-barrington Oct 1, 2026
1b3518c
HTTP/2 to peers over several connections (cluster.internal_http2_conn…
austin-barrington Oct 1, 2026
3fbdad1
Admit 32 writes per CPU by default
austin-barrington Oct 1, 2026
a7012ef
Peers always talk HTTP/2
austin-barrington Oct 2, 2026
225f039
Test that DROP DATABASE removes its series metadata
austin-barrington Oct 2, 2026
4b8197d
Bound flush memory: chunk points and concurrent chDB inserts
austin-barrington Oct 3, 2026
c81ab1e
heap-profiling feature (jemalloc profiling)
austin-barrington Oct 3, 2026
15b5245
Sweep: keep databases between points by default (SWEEP_DROP)
austin-barrington Oct 3, 2026
b27c4e3
Cap region transfer chunks at 50,000 points
austin-barrington Oct 3, 2026
6afe78c
Bound WAL reads by bytes so a restart's first flush can't OOM
austin-barrington Oct 3, 2026
1e57f0b
Sweep waits for every region's series count before measuring
austin-barrington Oct 4, 2026
d062a53
Run region moves in parallel, paced on the sender, off the tick
austin-barrington Oct 4, 2026
36eafa8
Add a linear-scaling acceptance test with an exact read-back
austin-barrington Oct 4, 2026
a37e8e7
Hint a spilled replication batch to its region's replicas only
austin-barrington Oct 4, 2026
5507338
Acceptance test records DB cores and CPU per point for every run
austin-barrington Oct 5, 2026
3218e7d
Acceptance test holds host load equal across N
austin-barrington Oct 5, 2026
5ff08ff
Acceptance CPU per point from cgroup counters around the measure phase
austin-barrington Oct 5, 2026
b772792
Store the WAL uncompressed
austin-barrington Oct 5, 2026
9f9434c
Move local points into the WAL append instead of cloning them
austin-barrington Oct 5, 2026
2db7e22
Compute a point's series id once and cache it on the point
austin-barrington Oct 5, 2026
b851ce5
Hash the ingest caches with foldhash and check them once per batch
austin-barrington Oct 6, 2026
933deda
Check tag values only for points of new series
austin-barrington Oct 6, 2026
fc3f5ae
Encode point frames and WAL entries with bincode in one pass
austin-barrington Oct 6, 2026
77c044f
Add a hidden bench-read subcommand for the read path
austin-barrington Oct 6, 2026
77c24ad
Filter fact rows before coalescing: push time and tag equality down
austin-barrington Oct 6, 2026
c565d63
Give region queries sent to peers the query timeout, not 5 s
austin-barrington Oct 6, 2026
41a4d1c
Bound every user and region query with ClickHouse's own time limit
austin-barrington Oct 6, 2026
00a420b
Push tag regex and OR-of-tag selections down to the fact rows
austin-barrington Oct 6, 2026
0258365
bench-read: dashboard query shapes
austin-barrington Oct 6, 2026
84f53f0
Join only the selected series when a tag selector is pushed down
austin-barrington Oct 6, 2026
b47df83
Time the stages of a sharded query
austin-barrington Oct 6, 2026
0cb7af4
Create fact tables with 1024-row granules
austin-barrington Oct 6, 2026
12a7fda
Revert "Create fact tables with 1024-row granules"
austin-barrington Oct 6, 2026
9150ef2
Skip the series join when every tag the query uses was pushed down
austin-barrington Oct 6, 2026
a36e668
Full e2e run: write and read latency, and the cost of scaling
austin-barrington Oct 8, 2026
a4328e1
Replay hints for peers that never went away
austin-barrington Oct 8, 2026
16abbf2
Keep region copies out of the scheduler tick and let them finish
austin-barrington Oct 8, 2026
495f6e9
e2e report: an empty map is settled
austin-barrington Oct 8, 2026
e87702e
Region copies read only the unflushed part of the WAL
austin-barrington Oct 8, 2026
95196f8
Merge regions outside the scheduler tick
austin-barrington Oct 8, 2026
78bf5dd
Start moves whenever a mover slot is free
austin-barrington Oct 8, 2026
46eb8a5
Merge only regions on the same peers, and move no data
austin-barrington Oct 8, 2026
1cfe61c
Accept date-time strings for time in WHERE
austin-barrington Oct 8, 2026
e59f5c1
Keep routing to decommissioning nodes until their regions move off
austin-barrington Oct 9, 2026
0815180
Treat refused shard map ops as errors; merge only equal epochs
austin-barrington Oct 9, 2026
5b50db2
Check only the oldest WAL entry for the legacy format on open
austin-barrington Oct 9, 2026
d4177b1
Keep only a margin of WAL behind the flush in a sharded cluster
austin-barrington Oct 9, 2026
3ec8209
Show why a coalesced forward write failed
austin-barrington Oct 9, 2026
f67595b
Sharded nodes sync metadata and the shard map, not a peer's WAL
austin-barrington Oct 9, 2026
985e4ce
Sharded materialized views roll up only rows ingested on the node
austin-barrington Oct 9, 2026
dba63b7
DROP DATABASE deletes metadata by range, and only its own
austin-barrington Oct 9, 2026
2fef3cb
Bound the sharded WAL margin in bytes, not entries
austin-barrington Oct 10, 2026
febb810
Export of a measurement unknown on this node is empty, not an error
austin-barrington Oct 10, 2026
ca7bbb8
Bootstrap a replicated view's shard spaces after the Raft apply
austin-barrington Oct 10, 2026
dcd0e6b
Send the source schema with a replicated CREATE MATERIALIZED VIEW
austin-barrington Oct 10, 2026
83065c2
Route literal regex and OR tag filters to their regions
austin-barrington Oct 10, 2026
d817067
Fetch a view's source schema from region owners when the node has none
austin-barrington Oct 10, 2026
d6f0d2c
Read view partials from every node and stop moving destination regions
austin-barrington Oct 10, 2026
777909d
Default the chDB query pool to 12 connections
austin-barrington Oct 10, 2026
a7b8cfc
Drop rows a view destination refuses instead of holding the WAL
austin-barrington Oct 10, 2026
af4d83a
Backfill a sharded view's history once, from the leader
austin-barrington Oct 10, 2026
26e0101
Cut a copy's chunk from the fact table before joining its series
austin-barrington Oct 10, 2026
c4d4f17
Run placement steps that copy nothing without a mover slot
austin-barrington Oct 10, 2026
959370b
docs: what to expect when operating a sharded cluster
austin-barrington Oct 10, 2026
89c84e8
Place a new measurement's regions only on nodes that are staying
austin-barrington Oct 10, 2026
b201f53
docs: rollups live on the ingesting primary; scale-down, membership a…
austin-barrington Oct 10, 2026
701af07
Never copy a view destination's rows when a region changes hands
austin-barrington Oct 10, 2026
290e802
Hand a departing node's view rollups to a node that stays
austin-barrington Oct 10, 2026
27baa43
docs: query pool sizing, region count cost, backfill and rollup handoff
austin-barrington Oct 10, 2026
837b224
Let a departing node retire while view destinations still name it
austin-barrington Oct 10, 2026
133c992
docs: verified rollup handoff through 3 -> 6 -> 3; departing node ret…
austin-barrington Oct 10, 2026
1f28302
Size the query pool from the CPUs and let SET change it at run time
austin-barrington Oct 10, 2026
fc38ca8
Release 0.9.0-beta
austin-barrington Oct 10, 2026
44cf326
docs: 0.9.0-beta release notes, operator versions, runtime settings
austin-barrington Oct 10, 2026
39b42f9
Compress the WAL again: uncompressed it saturated the disk
austin-barrington Oct 11, 2026
b494ebd
docs: final beta numbers, disk saturation and space after a scale-up
austin-barrington Oct 11, 2026
90c218c
Runtime settings: 19 settings change live with SET
austin-barrington Oct 11, 2026
2c7c0a2
Runtime settings: plain values in errors; n3 results in the operating…
austin-barrington Oct 11, 2026
baf3d29
Cluster auth and GRANT READ / WRITE
austin-barrington Oct 11, 2026
7346850
Keep grants when a user changes or syncs; valid Raft timing defaults
austin-barrington Oct 11, 2026
fef907f
docs: cluster auth, grants and Raft timing; UAT behaviour
austin-barrington Oct 11, 2026
499c705
Shard merge: linear in rows, columns matched by name
austin-barrington Oct 12, 2026
006644f
docs: shard merge timing and operator rollout time on UAT
austin-barrington Oct 12, 2026
192ce92
Faster wide Grafana panels: gzip, no region sort, merge fast path
austin-barrington Oct 12, 2026
4bef597
Aggregates keep answering while a region primary restarts
austin-barrington Oct 12, 2026
7e656c6
SHOW TAG VALUES applies WHERE, LIMIT and OFFSET
austin-barrington Oct 12, 2026
2ae8b2f
Aggregate time is the WHERE's lower bound, as in InfluxDB
austin-barrington Oct 12, 2026
445c928
Set TCP_NODELAY on accepted connections
austin-barrington Oct 12, 2026
9461ca3
PERCENTILE/MEDIAN on sharded data; SELECT column order; no series_id
austin-barrington Oct 12, 2026
b44e5c0
docs: query latency against InfluxDB 1.11 on UAT
austin-barrington Oct 12, 2026
430b5a0
One region per new measurement; small neighbours co-locate and merge
austin-barrington Oct 12, 2026
0e5425a
docs: one region per new measurement, co-locate and merge
austin-barrington Oct 12, 2026
d12d0c2
Region copies: no rows skipped for long series, smaller chunks on 413
austin-barrington Oct 12, 2026
45d962a
docs: region copy fixes
austin-barrington Oct 12, 2026
f17766f
docs: UAT latency after regions collapsed
austin-barrington Oct 12, 2026
1371c63
Python client for HyperbyteDB
austin-barrington Oct 12, 2026
505d8be
Release 0.9.1-beta
austin-barrington Oct 13, 2026
73a7824
Tests: aggregate time is the WHERE's lower bound; gzip test client
austin-barrington Oct 13, 2026
a98fa33
Refuse a cluster token peers cannot send; release regions on panic
austin-barrington Oct 15, 2026
a759fc6
Review: status from a peer, lane bound, scheduler off two adapters
austin-barrington Oct 15, 2026
18ec0db
Ports: no defaults that silently succeed
austin-barrington Oct 15, 2026
049b562
Forward queue: a port, O(log n) indexes, I/O off the async workers
austin-barrington Oct 15, 2026
32667df
Move serve out of the application layer
austin-barrington Oct 15, 2026
236f18a
Cluster token: injected ClusterAuth instead of a global
austin-barrington Oct 15, 2026
150aac4
SET cardinality limits no longer refuse transfers and replicas
austin-barrington Oct 15, 2026
b0a225a
Runtime settings: one RuntimeSettings per node, not a global
austin-barrington Oct 15, 2026
8923faf
docs: cardinality limits set at run time apply to client writes
austin-barrington Oct 15, 2026
8f5f535
e2e report: a timed-out read-back is not lost points
austin-barrington Oct 16, 2026
691d170
Region copies: read chunks in index order, send them raw
austin-barrington Oct 16, 2026
805d21a
Region copies: read the next chunk while sending this one
austin-barrington Oct 16, 2026
5e622dc
Review fixes: health answers 503, newlines in string fields, R7 gate
austin-barrington Oct 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,4 +117,7 @@ Desktop.ini
.cursor/
hyperbytedb-operator
chdb-rust
docs/audits/
docs/audits/
# Python
__pycache__/
integrations/checkmk/dist/
Loading
Loading