Repository navigation
HyperbyteDB 0.9.1-beta - #116
Merged
Merged
Conversation
Bootstrap now takes RF = min(configured, live members), so n=1 gets peers=[self] even when configured RF is 3.
A joiner's map_version must match the cluster's before any peer-set change, so staging cannot run against a lagging epoch.
A joining node became a member but never received data for measurements that already existed: `try_split` clones the parent peer set and `try_rebalance` only moves a primary among peers already in the region, so an existing region's peer set never grew to include a new process. Add `ShardMapOp::AddPeer` and a scheduler placement step that stages the region onto a live Active member before committing it as a peer. Staging first is the contract here — the heal path's commit-then-stage ordering would publish a peer that holds no rows. Apply refuses `AddPeer` on a region carrying outstanding transfer debt, on a duplicate peer, and on a stale epoch. `ShardRehomeRequest.stage` lets the scheduler ask a remote primary to run the same staging push when the leader is not the region's primary; the rehome handler skips `complete_region_transfer` for a staging push since ownership does not change.
A joiner that received region data was still only a replica: nothing moved a primary onto it, so writes kept landing on the original owner and the new process added HA but no write capacity. Add two placement steps to the scheduler tick, both stage-then-commit so an empty node never owns a range: - `try_place_primary` hands a region's primary to its newest peer while that peer carries strictly fewer primaries than the current one. The "newest peer" bias is what makes it terminate — once the joiner is primary the rule no longer applies, so ownership cannot oscillate back, and the count guard stops every region stampeding onto one node. - `try_place_idle_member` covers 3->4, where no region is under RF and `AddPeer` correctly declines. An existing replica steps aside for the newest member via `MovePeer`, which appends it and so makes it the next primary-placement candidate. `try_rebalance` scored a peer with no heartbeat row as 0 bytes, which made any freshly staged replica look infinitely lighter than the primary and handed it ownership on the strength of missing telemetry. Unmeasured peers are now skipped rather than treated as empty. Integration coverage asserts the contract rather than the midpoint: a write aimed at the replica-only joiner forwards and leaves its WAL empty, and the same write applies locally once the primary has moved.
Both placement steps ran against the region as it looked when the tick snapshotted the map. A placement earlier in the same tick invalidates that: `AddPeer` bumps the epoch and appends a peer, so the next step would stage an entire region copy and only then have its proposal rejected by the epoch CAS — wasted transfer, repeated every tick until the snapshots happened to line up. Re-read the region from a fresh snapshot inside each step, and skip it if the region has since been split, merged, or dropped.
Seven findings from reviewing the branch, four of which could corrupt data or wedge a cluster. **Rollup destinations were eligible for placement.** `apply_transfer_push` has no idempotence guard, so a region staged onto a joiner whose map op then fails is staged again next tick. `ReplacingMergeTree` measurements collapse the redelivery; `SummingMergeTree` rollup destinations sum it and are corrupted permanently. `enqueue_reconciliation` already excluded these spaces for exactly this reason — all three placement paths now share that guard, failing closed when the lock is contended. **`AddPeer` had no upgrade gate.** `ClusterRequest` is serialised into the Raft log, so a committed op an un-upgraded voter cannot deserialise wedges it. `ClearVerified` set the precedent; `AddPeer` now has the same gate and defaults **off** for the release that introduces it. **`sync_shard_map` could move the map backwards.** The sync peer is whichever Active node came first out of a HashMap — not the leader, not necessarily the most-applied node. Installing an older snapshot dropped ops Raft never redelivers, because `replace_map` bypasses the state machine and leaves `last_applied` untouched; every later op then failed StaleEpoch and the node silently stopped owning regions it held data for. Install only a strictly newer map. **A missing shard map aborted the whole join.** `/internal/shard/map` only exists on peers with sharding enabled, so a 404 from an older or unsharded peer failed `join_and_sync` *before* WAL catch-up; after the retries the node went Active having synced nothing. It is best-effort now: with no map there are no regions to move, and the leader's own catch-up check still gates placement onto this node. Also: `try_place_live_member` never re-read its region, so it staged a full copy before failing the epoch CAS (the sibling fix missed it); and the steps after a placement ran on the region as it looked *before* that placement committed — `try_rebalance` would spend an entire region copy on a stale peer list. A placement that commits now yields the region. Placement was reachable only from `tick()` and no test called it, so deleting those calls left the suite green. Adds a harness with a mock peer that drives a real tick, covering both the placement and the upgrade gate.
Placement could stop cluster-wide, permanently and quietly. `joiner_map_caught_up` required exact `map_version` equality and `live_replica_candidate` always returned the lowest-id non-peer, so a single node that never matched was chosen first, failed the probe, and was chosen again the next tick. No other candidate was ever tried, for any region, and the only trace was a `debug!` line. - Catch-up accepts a candidate that is *ahead*. The cluster version comes from a snapshot read just before the probe, so a node that applied an op in between is legitimately ahead and was being rejected for it. Behind is still refused — that is what the gate is for. - `live_replica_candidates` returns every eligible member in id order and placement takes the first that passes its probe, instead of stopping at a candidate chosen before any probe ran. - A region that goes `PLACEMENT_STALL_WARN_TICKS` ticks with candidates but none caught up now warns and increments a counter. A stall that only shows up at debug level is a stall nobody finds. - `/internal/shard/map` serialises every space and region to answer one `u64`, and it was probed once per region per tick — O(regions x map size) bytes on every tick of a join. Probes are cached per peer for the tick. Also corrects the `handle_shard_transfer` staging comment, which still claimed a local-measurement guard that P1.3 deliberately removed. H.6 (whether the gate should compare the Raft applied index instead) is answered in the plan, no code change: the index is already exposed at /internal/raft/metrics, but `replace_map` installs a map without advancing it, so an applied-index gate would reject exactly the joiner it should admit. Fix belongs with Phase 2's snapshot-based map recovery.
`try_place_idle_member` was documented as giving "a member that joined an already-replicated cluster" a replica slot, which reads as join-triggered. It is not. The target is the active member with the greatest `joined_at` — whoever joined last, with no recency window, so "last" may mean months ago. The rule fires whenever that member is under-loaded against some region's non-primary peer, which means a stable-but-unbalanced cluster starts moving replicas on the first tick after an upgrade rather than on any join. The behaviour is deliberately unchanged. It is bounded and convergent, and every copy is staged and verified before the map changes, so this is a scheduling surprise rather than a correctness risk — acceptable while sharding is beta. Gating it was considered and rejected for now; the doc says so, and says to revisit before sharding graduates. Renames `newcomer` to `latest_member` throughout, so the identifier stops implying recency too. No logic change.
Shipped it off, which was the wrong call. The gate is an opt-out for a rolling upgrade, not an opt-in for the feature — `transfer_clear_proposals_ enabled` guards the identical hazard and defaults on, and defaulting off leaves region placement inert for everyone who enables sharding and never finds the flag. The hazard is bounded: a follower on a build that does not know `AddPeer` rejects the whole append RPC (axum fails the body before the handler runs), so it stops replicating and the leader can lose commit quorum for the duration. It resolves as soon as every node is upgraded, and ingest does not go through Raft. Operators disable the gate for the rollout window.
Placement must agree across every node, including nodes on different builds mid-rollout, so this cannot use DefaultHasher -- std's hasher is explicitly not stable across releases. SplitMix64 constants live in our own source and are frozen; the avalanche test guards against anyone retuning them.
Add before Promote before Remove, one step per call. The ordering is load-bearing: adding before removing is what keeps RF from dipping while converging, and promoting before removing keeps the primary off the node being evicted -- which is also what gives drain correct behaviour without a rule of its own. Carries forward the transfer_outstanding guard both deleted candidate functions had. AddPeer apply rejects while debt is outstanding, so proposing anyway spends a full region copy on a guaranteed rejection every tick, uncounted against the movement budget. repeated_steps_converge_to_a_fixed_point is the convergence argument itself: with a pure target function, termination needs no cooldown and no recency bias.
different_starts_spread_across_nodes is the point of the whole phase: a measurement's regions must not all land on the same RF nodes. Today try_split clones the parent peer set, so 48 regions use three nodes; with this they use most of twelve. Rendezvous hashing is chosen for its movement property, which the other two tests pin: adding a node moves a minimal share rather than reshuffling, and removing one moves only the regions that node held. Ranking by raw hash is the weighted score w/-ln(h/MAX) under uniform weights, since that is a monotonic transform. A weighted variant must switch to the full score.
The mirror of AddPeer, and what makes drain and rebalancing possible at all: until now a region's peer set could only grow. Refuses to remove the primary or the last peer so a region is never left ownerless, and carries AddPeer's stale-epoch and transfer-debt guards. No production match on ShardMapOp swallows the new variant -- checked, since a catch-all would have made this compile while doing nothing.
Both default to the safe reading: RemovePeer proposals on, matching AddPeer, and a movement budget of 0 meaning unlimited so the throttle never engages until someone asks for it.
Pure refactors with no behaviour change, so convergence can reuse both rather than duplicating a two-branch staging path and a probe loop that exists to stop one lagging node blocking every region. first_caught_up keeps the original accounting exactly: an empty candidate list clears the stall rather than recording one, since having no candidates is not the same as being stalled on a lagging peer.
Generates its own load rather than delegating to sharding-perf-ingest.sh, which hardcodes 100k series into measurement "metrics" and honours only API and DB. Driving that with SERIES=5000000 would have ingested 100k series into roughly one region at every node count and reported a flat curve -- a false negative on the whole design, arriving as a verdict. Warns when regions < nodes, since a flat curve there is the cardinality floor rather than a placement defect. The region-count probe is checked against the real ShardMapJson shape: the leading quote in "region_id" keeps it from also matching next_region_id.
NodeState answers three independent questions -- can it serve, should it
hold regions, must its data be evacuated -- and the code tested == Active
for all three. That is what produced the same placement defect twice. The
questions are now three named predicates, and no ordering of a single
state has to satisfy all of them: a draining node serves nothing yet
keeps its regions, a disconnected one keeps them too, only a departing
one loses them.
transition() refuses moves out of a terminal departing state. set_state
overwrites unconditionally, so the preStop hook's /internal/drain could
put a decommissioning node back into Draining mid-evacuation, re-admitting
it as a placement candidate while its regions were still moving off.
Draining now recovers to Active on a ready self-report. It previously sat
in the never-promoted list, so a node that restarted after a preStop drain
stayed Draining in its peers' views forever -- broken today, and
load-bearing once drain keeps its seat. operator_states_are_never_promoted
encoded the old contract; it now covers Joining, Decommissioning and
Leaving, and the new behaviour has its own test rather than a weakened
assertion.
The compiler found exactly one exhaustive match. write.rs had a `_ => {}`
that would have silently admitted client writes to a node being removed;
its arms are now explicit so the next variant cannot slip through the
same way.
One branch carries the spec, the phase-1 plan, the sharding test plan and the code, so the design and what was built from it stay reviewable together.
Variant order is the sort order and that is the point: durability must never queue behind balance. RF violations and evacuation are exempt from the movement budget; only convergence yields. A region at full peer count with a dead peer is under-replicated in substance even though peers.len() still reads RF, so move_tier takes the unhealthy set rather than trusting the count. The middle tier is Evacuation, not Drain -- under the new membership model drain is a restart that moves nothing, and only decommission evacuates. Lives in domain/sharding/placement.rs beside the rest of the pure policy rather than in the application-layer scheduler.
Placement is now a pure function of membership, so a demotion moves data. One dropped probe must not reshuffle a node's regions. Demotion needs cluster.heartbeat_miss_threshold consecutive misses -- a knob that already existed, defaulted to 5, was already in the CRD, and until now was read only by flush_service while the prober demoted on the first failure. The counter lives on NodeInfo, not in probe_peers. probe_peers is stateless and called fresh each tick, so a map local to it would reset every tick and pin the count at 1 -- hysteresis that compiles, passes its unit tests, and demotes on the first miss exactly as before. A success decrements rather than resets. Hard-reset never demotes a node alternating four misses and one success: unreachable 80% of the time yet permanently Active, which is worse than the rule it replaced. The Unreachable arm previously bypassed decide_transition with its own inline check; both arms now go through it, so the hysteresis cannot be sidestepped. unreachable_peer_is_disconnected encoded the pre-hysteresis contract and is rewritten to walk to the threshold rather than widened with arguments that would have quietly inverted it.
A measurement's first region starts at 0, so bootstrap placement is exactly target_placement(key, 0, members, rf). Computing it any other way means a freshly bootstrapped region is immediately misplaced and the scheduler moves it on the first tick. The previous ordering used DefaultHasher, which std does not guarantee stable across releases. Placement is recomputed continuously on every node, so two builds disagreeing would scatter regions during a rolling upgrade -- the same hazard the frozen hash exists to avoid.
The tick now moves each region one step toward target_placement's output and the three recency-keyed rules are gone. Their difficulty was proving termination without a cooldown, which is why primary placement keyed on the newest peer and idle placement fired on max(joined_at) rather than on an actual join. With a pure target function that argument disappears: each step strictly reduces the difference and next_placement_step returns None at the fixed point. Add probes every target member the region is missing, not just the one next_placement_step names. PR #115's placement_skips_a_lagging_candidate test caught this: committing to a single peer before probing let one lagging node block that region forever -- the bug review findings #9/#10 fixed, which I had reintroduced. All missing members belong in the region, so any may be staged first. Chunks C3 and C5 merged because -D warnings will not accept a method reachable only from cfg(test): convergence could not land unwired.
Drain and decommission shared one code path while meaning opposite things. shard_handoff proposed TransferPrimary AND MovePeer and passed drop_source: true, so every restart evicted the node from every region it led and deleted its local copy -- and the preStop hook runs that on every pod. Now only the crown moves: the seat and the data stay, so a restarted node resumes with nothing to re-stage. Decommission sets Decommissioning, hands off primaries, flushes, and returns. It never declares Leaving; the evacuation loop is the only thing that may, once a replacement peer set is actually established. Lifecycle changes go through transition(), which refuses to downgrade a terminal state. Without it the preStop hook's /internal/drain would clobber a decommissioning node back to Draining and re-admit it as a placement candidate mid-evacuation. handle_leave's self-leave arm had the same hazard and is fixed with it. The shutdown path can no longer read "already drained" off membership, since a successful drain now ends at Draining rather than Leaving, which is indistinguishable from never having drained. DrainService tracks it directly.
This is the defect the phase exists for. try_split cloned the parent's peer set, so every descendant region of a measurement lived on the same RF nodes: splitting a hot measurement 128 ways still used three. The right child is now placed by the hash, keyed on start rather than region_id because Split allocates the child's id at apply time and a proposer cannot hash on an id that does not exist yet. The left child keeps the parent's start, so its placement is identical by construction and its rows never move -- an eager split costs only the right half. The right child is published holding its primary ALONE, not the full target set. try_split stages to right.primary only, so committing the whole target would publish RF-1 peers with zero rows; scatter falls back to replicas and that reads as silently empty results. It would also be self-sealing, since a region whose peers already equal its target is read as converged and never backfilled. Convergence grows it instead, in the unthrottled RfViolation tier.
The allowance is sized from the scheduler's real tick cadence, captured in run(), not from split_merge_interval_secs. The latter is an unrelated split/merge cooldown defaulting to 3600 while the scheduler ticks on heartbeat_interval_secs, default 10 -- sizing from it would hand every tick an hour's allowance and leave the throttle 360x too loose. The gate sits after the converged check, never before. Gating first would skip the None branch for already-converged regions once a tick's budget was spent, leaving their stall counters uncleared and reporting healthy regions as stalled under exactly the load stall detection exists for. RF violations and evacuation are exempt: safety never queues behind balance. A region at full peer count with a dead peer counts as an RF violation, since peers.len() alone would call it merely misplaced. tick_moved_bytes is a bare AtomicU64 -- the scheduler is already shared behind an Arc -- and Relaxed is correct, since only the scheduler task touches it and it orders no other memory.
decommission_complete is the only thing that may write Leaving for a decommissioning node. drain reaches its terminal state after a 90s-capped replication wait, which says nothing about whether any region moved -- and the operator is meant to wait for Leaving before deleting the node's PVC, so resolving it on a timer would race evacuation against volume deletion for exactly the nodes holding the most data. The sweep runs after the region pass so a node that finishes evacuating during a tick is retired in that same tick, and goes through transition() so it cannot move a node that is no longer decommissioning.
A dead peer is dropped first rather than swapped. There is no live source to stage from, so preserving its membership only delays RF restoration -- the deliberate asymmetry with drain, which has a live source and therefore restores before it removes. This also drops the hand-rolled replacement selection. Picking the least-loaded active member could choose a node the placement function disagrees with, which convergence would then move again; removing the dead peer instead leaves the region below effective RF, where convergence classifies it RfViolation -- top tier, exempt from the budget -- and stages whoever the hash actually wants, before committing rather than after. RemovePeer refuses the primary, so a dead primary is left to try_failover_unhealthy_primary, which runs immediately after this in the tick; the peer is removed on a later pass once it no longer owns the region.
The six free functions the old rules used were dead in production after the switchover, but three of PR #115's integration tests still called them directly -- computing what the retired policy would decide and applying the op by hand, never going through tick(). They were green while exercising code nothing called, which is worse than red. Two of them needed a measurement rename rather than a weakened assertion. Under hashing p14db/cpu targets node 1 first, while the test body is built around node 2 becoming primary; p14db/disk targets [2, 1], so it is a genuine P1.4 case again. Same for the 3->4 test, where p16db/disk targets [4, 3, 2]. Two assertions changed meaning, deliberately. "RF is satisfied so AddPeer must not fire" is now the opposite: a fourth node needs no idle-member rule, it simply enters the targets its hash wins. And "the primary is never displaced" -- which the old rule guaranteed outright -- became a walk of the steps asserting no Remove ever names the sitting primary, since Promote is ordered before Remove. The new policy may displace a primary, but never while it still holds the role. 329 lines of policy deleted; the behaviour it implemented is now a pure function and a diff.
Two bugs, both mine, both invisible to 850 in-process tests and both caught by a real 6→5 scale-down. First run lost 1196 of 5000 points, second 625; this passes with 5000 intact and replication factor restored. is_active_peer tests `== Active`, so a Decommissioning node read as dead and the drop-first heal path removed it without staging its rows anywhere -- deleting the map's only reference to that copy. C8 argued drop-first is right because a dead node has no live source to copy from; that is true of a disconnected node and exactly wrong for a departing one, which is alive and still holding data. Departing peers are now excluded from dead-peer detection: convergence owns them, and it stages before it removes. The deeper one was a trap I built both halves of. C4 published a split's right child holding a single peer, and split children carry transfer debt, and the transfer_outstanding guard makes convergence skip debt-carrying regions -- so the child was stuck below replication factor with its rows still on the parent's primary, unreadable and unrepairable. With regionSplitSeries at 25 against 5000 series that was the steady state, not an edge case. The child now keeps the parent's peer set, whose nodes replicated the parent's whole range and therefore already hold its rows; only the primary moves eagerly, and only to a peer that has the data. Convergence walks the child to its hash target afterwards, staging each peer before committing it. The e2e launcher also waited on pod readiness rather than job completion, so it copied the report before the test had written one and reported a failure that had not happened.
Two ways the launcher could call a run green without evidence. It waited on pod readiness rather than job completion, so it copied the report seconds after the container started -- before the test had written one -- and reported a failure that had not happened. That was the first "failure" of this branch's e2e. Worse in the other direction: kubectl cp failures are swallowed, and the launcher's own exit trap writes a report with an empty goals table saying "Overall: PASS". So a failed copy left a stale or vacuous report in place and the PASS grep read it. A run whose report never arrived would be reported as passing. The launcher now waits for the pod to reach a terminal phase, clears its own goals so it cannot emit a verdict it did not earn, removes any prior report before copying, and refuses to call a run passing unless the copied report actually contains gate rows.
Found collapsing UAT's regions: every copy of Telegraf's cpu regions was refused with 413 and retried forever, so their learners never promoted. - A copy paged through chDB on series_id alone, re-reading the boundary series from its start. A series with more rows than one chunk (50,000) never advanced, and the rest of it was skipped with a warning -- rows lost on every move of a series older than ~6 days at 10 s. Paging is now on the table order, (series_id, time, ingest_seq), strictly after the last row read, so a series of any length is copied whole, and two versions of one point split across chunks both go. - Chunks are capped in points and the receiver's limit is in bytes; a wide measurement (cpu: 10 fields and tags) exceeded it. A 413 now halves the chunk for the rest of the copy (down to 100 points) and sends the same rows again.
Standard library only, Python 3.9+. Batched, gzipped writes with backoff that honours Retry-After, 413 split, typed errors; InfluxQL queries with bound params. Unit tests against a fake server and integration tests against a real one (HYPERBYTEDB_URL).
0.9.0-beta was tagged but never published; 0.9.1-beta replaces it. Release notes renamed and updated (operator v0.3.0, Python client, two known issues).
The compat test still expected epoch 0 after 8130d40. The gzip test failed in a workspace build: the CLI enables reqwest's gzip feature, so the test client decompressed and dropped Content-Encoding.
PR SummaryHigh Risk Overview Sharding and operations: The release targets linear write scaling and safe scale-up/scale-down (decommission/evacuation), with extensive kind-based e2e scripts (lifecycle, continuous writes during topology changes, splits under load, rolling restart) plus new cluster manifests (1/3/6-node sharded profiles, 12-worker linearity layout) and RBAC for in-cluster test jobs. Docker/kind setup gains optional Server-facing capabilities (per release scope): materialized views on sharded clusters, cluster auth ( New Python client ( Docs add linear-scaling design, sharding real-world test plan, scale-down runbook, and user-guide updates (runtime settings, MV/CQ column naming, operating sharded clusters). CLI adds bench modules; Reviewed by Cursor Bugbot for commit 728374f. Configure here. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 728374f. Configure here.
austin-barrington
force-pushed
the
release/v0.9.1-beta
branch
from
October 13, 2026 21:45
728374f to
73a7824
Compare
A cluster token that is not a valid header value (a Secret written with echo ends in a newline) was silently left off every peer request, so every internal route answered 401 and Raft never formed. Config validation now refuses it at startup, without showing the value. A scheduler task that panicked or was cancelled kept its region claimed until restart, and a panic in a promotion or merge batch left its busy flag set, stopping all later ones. Region claims are now a guard that releases on drop, and the busy flags clear on drop.
A rehome status that is not a u16 is a 502, not a truncated code. The forward lane comment says why its channel needs no bound. The rehome progress header moves to the domain's wire types and the map-version probe goes through PeerClient, so the scheduler no longer imports the HTTP handlers or the sync client.
pending_forwards defaulted to 0 (safe to retire), set_setting to a no-op and flushed_margin_start to "keep nothing". Every adapter overrode them, but a new one that forgot would compile and lose writes or settings. They are required now; the single-node ingestion service says 0 itself.
PeerIngestionService now holds a ForwardQueuePort, and ForwardEntry is a domain type, so the application no longer imports the adapter. The adapter runs its RocksDB calls on the blocking pool, as the WAL does. claim and the per-write lag check walked every entry under a lock every write takes; ready, delayed and by-age indexes make both O(log n) with the same order. complete now takes the retry and writes both in one batch: the queue never holds a batch and its retry at once, even across a crash. A failed complete releases the entries instead of holding them until restart.
application/runtime::serve built the services, started the tasks and built both HTTP routers: top-level wiring, now in src/server.rs next to bootstrap. application/runtime keeps the control runtime, region write stats and the region heartbeat reporter.
The token lived in a process-wide static that every peer client read when built and the auth layer read on each request. ClusterAuth is now built once at startup from auth.cluster_token (refusing one that cannot be a header, or an HTTP stack that cannot start) and handed to PeerClient, the sync client, the Raft network, AppState and the query service. The application builds peer clients through a PeerHttp port, so it no longer imports cluster_auth; a client that fails to build is logged, not silently swapped for one without the token. Raft proposals for shard map ops reuse the node's pooled peer client instead of building a client per call. Network::default, which had no credential, is gone.
effective() replaced whatever limits a caller passed with the SET override, including the unlimited ones region transfers, sharded replicas and sync pass for data that already exists. After SET max_tag_values_per_measurement = N, moving or replicating a measurement over N was refused. IngestCardinalityLimits is now a shared handle: the client-facing limits built at startup get the SET through appliers, every service holds a clone of them, and unlimited() limits are never touched. The single-node ingestion service takes the shared limits instead of building its own. Reading them is two atomic loads instead of a registry lock and lookup.
The SET registry was a process-wide static. Nodes run in one process (the sharded test harness) shared it, and register_applier replaced the previous node's applier, so a SET resized only the last node's pools. RuntimeSettings is now built in build_services and handed to AppState, the query service, flush service, scheduler, routing context, Raft store and sync client. The query service refuses SET and SHOW SETTINGS without settings rather than changing nothing; elsewhere a SET without them is stored and applies at the next start (logged). The log filter is per process, so init_logging returns a LogLevel that serve registers with the node's settings.
The steady writer now writes 195-303M points per scaling step, and a count over that outlasts the 30 s query timeout; the read-back read the timeout as zero and reported every steady point lost. It now raises query_timeout_secs for the read-back, reports a failed count as unreadable, and logs every layout poll (with staging learners and node count) to show why a layout does not settle.
Each chunk was cut with ORDER BY (series_id, time, ingest_seq), which is not the fact table's sort key, so ClickHouse sorted every remaining row of the region to return one chunk: 3.7 s a chunk on a 280M-row region on n3, and copies ran at 0.3-0.5 MB/s on 2.4 cores. Chunks now order by the sort key with LIMIT ... WITH TIES (every version of a point stays in one chunk) and resume after (series_id, time) spelled out so the index narrows it: 0.13 s a chunk. Pushes carried the line protocol inside JSON, one number per byte: 3-4x the bytes to write, send and parse. They now go to /internal/shard/transfer/raw with the body as is and the rest of the payload in a header; a node without the route answers 404 and gets the JSON push.
Export, push and the receiver's apply ran in turn, so the sender sat idle through every apply. The next chunk is now read while the current one is pushed; a chunk refused as too large discards the read-ahead and re-reads that range in smaller chunks.
- Python client: health() returns a 503 (draining, syncing) at once; it was retried with backoff, about 15 s before returning the same answer. - Python client: a newline in a string field value is refused, as in names and tags; written raw it split the line. - Rolling-restart e2e: an unreadable RemovePeer counter fails R7 instead of reading as 0, which passed after a mass eviction. - Release notes: copies 20-30x faster; known issues found on n3 (series-only splits, replicas diverging after a scale-down under writes, the 30 s request timeout capping queries).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First public 0.9 beta. It replaces
0.9.0-beta, which was tagged locally but never published. Release notes are indocs/user-guide/release-notes-0.9.1-beta.md. Use it with operator v0.3.0 (hyperbyte-cloud/hyperbytedb-operator#16).Changes
GRANT READ | WRITE | ALL;SET <setting> = <value> | DEFAULTandSHOW SETTINGS.SHOW TAG VALUES WHERE;clients/python.Upgrading from 0.8.x
Tests:
cargo fmt --check,cargo test --lib,cargo test --test '*'andcargo test -p hyperbytedb-cliall pass locally. The last commit fixes two stale tests; neither was a server bug.After merge: tag
v0.9.1-beta; the release workflow builds and publishes it.