feat(qwp): stop resending the full symbol dictionary on every message - #66

Merged

glasstiger merged 208 commits into

mainfrom

qwp-delta-symbol-dict

Aug 5, 2026

Commits on Jul 9, 2026

Delta-encode the QWP symbol dictionary

glasstiger
and
claude
committed
de86197
Contain QWP dict catch-up send failures in the reconnect loop

glasstiger
and
claude
committed
3092230
Resume the QWP symbol-dict write-ahead from the durable size

glasstiger
and
claude
committed
9f3f7bd
Guard the oversized catch-up entry terminal

glasstiger
and
claude
committed
09cd706
Persist the QWP symbol-dict batch in a single write

glasstiger
and
claude
committed
18dd545
Free the recovery-seeded dict mirror in close()

glasstiger
and
claude
committed
2a1a2d6
Pin the recovery test on the catch-up frame

glasstiger
and
claude
committed
e130da9
Cover the disk-mode full-dict fallback

glasstiger
and
claude
committed
2b78b0e
Truncate the torn tail when reopening the symbol dict

glasstiger
and
claude
committed
282ac09
Guard the sent-dict mirror against int overflow

glasstiger
and
claude
committed
754eb3d
Merge branch 'main' into qwp-delta-symbol-dict
bluestreak01
authored
01f251d
Parse the delta header once per frame on send

glasstiger
and
claude
committed
4f44fb3
Fix stale self-sufficient-frame comments

glasstiger
and
claude
committed
0f9d88f
Make the delta-dict tests deterministic and leak-checked

glasstiger
and
claude
committed
d89f14f
Cover the split-batch delta contract

glasstiger
and
claude
committed
27aad9e
Accumulate the tail of a partial-overlap delta into the mirror

glasstiger
and
claude
committed
034d8f3
Guard against persisted-dict duplication on a failed publish

glasstiger
and
claude
committed
41fddb1
Merge branch 'main' into qwp-delta-symbol-dict
glasstiger
authored
f9430bc
Merge branch 'qwp-delta-symbol-dict' of https://github.com/questdb/java-questdb-client into qwp-delta-symbol-dict
glasstiger
committed
9602440
Cover the reconnect catch-up ACK alignment

glasstiger
and
claude
committed
0ef96af
Cover the retriable catch-up send containment

glasstiger
and
claude
committed
ff79b72
Tidy catch-up comments and harden two edges

glasstiger
and
claude
committed
8fd9ad1

Commits on Jul 10, 2026

Fix delta symbol-dict recovery and reconnect bugs

glasstiger
and
claude
committed
8fcd835
Adopt recovered dictionary into the send mirror

glasstiger
and
claude
committed
fc8b4ba
Harden delta symbol-dict recovery and NACK gating

glasstiger
and
claude
committed
8dd7723
Persist symbol delta from frame, skip re-encode

glasstiger
and
claude
committed
8853d8f
Guard catch-up frame overflow, add biting tests

glasstiger
and
claude
committed
10a125c
Fix orphan-drain losing symbol-dict mirror seed

glasstiger
and
claude
committed
fa50e81
Fix symbol-dict durability doc; add recovery test

glasstiger
and
claude
committed
444199e
Discard a surviving symbol dict on fresh start

glasstiger
and
claude
committed
7a95653
Harden delta symbol-dict send and persist paths

glasstiger
and
claude
committed
6d14727
Tidy delta symbol-dict dead code and edges

glasstiger
and
claude
committed
fdd7141
Fix catch-up terminal on a homogeneous batch cap

glasstiger
and
claude
committed
2fdbbf9
Fix recovery id desync on UTF-8-colliding symbols

glasstiger
and
claude
committed
8cfd7bd
Harden symbol-dict resource teardown paths

glasstiger
and
claude
committed
0b4ab3f
Defer catch-up commit; fix stale delta comments

glasstiger
and
claude
committed
186ae10
Plug ensureConnected leak; consolidate varint helpers

glasstiger
and
claude
committed
fef7203
Add a per-entry CRC to the SF symbol dictionary

glasstiger
and
claude
committed
cbaf474
Document split-flush and dict-getter contracts

glasstiger
and
claude
committed
6cc1307
Stop split-flush stranding a deferred prefix

glasstiger
and
claude
committed
c065ad4
Harden persisted symbol-dict edge paths

glasstiger
and
claude
committed
40c6a28

Commits on Jul 11, 2026

Guard the symbol-dict reopen against a >2GB file

glasstiger
and
claude
committed
94e3b1d
Latch a terminal on sent-dict mirror overflow

glasstiger
and
claude
committed
6c6dc40

Commits on Jul 12, 2026

Guard torn-dict resume and skip split re-encode

glasstiger
and
claude
committed
ff2e841
Give the catch-up cap gap a settle budget

glasstiger
and
claude
committed
50fc218

Commits on Jul 13, 2026

Cover defensive dict guards and tidy test helpers

glasstiger
and
claude
committed
7880b7a
Fail clean on a total symbol-dict tear on resume

glasstiger
and
claude
committed
2ad8f50
Bound recovered symbol id to committed frames

glasstiger
and
claude
committed
76fe4f1
Null loaded dict entries after a failed truncate

glasstiger
and
claude
committed
084f93f
Test the catch-up NACK guard at the ack boundary

glasstiger
and
claude
committed
8facb8d
Cover the cap-gap budget reset on success

glasstiger
and
claude
committed
031777e
Cover file-mode split write-ahead dict persist

glasstiger
and
claude
committed
0c0b9dc

Commits on Jul 14, 2026

Cover split-preflight and torn-dict guard gaps

glasstiger
and
claude
committed
bbc01a8
Cover the persisted symbol-dict short-write path

glasstiger
and
claude
committed
39e9e52
Harden the symbol-dict fd close and split sizing

glasstiger
and
claude
committed
265b703
Recover full-dict SF slots without bricking build

glasstiger
and
claude
committed
2fa5cd9
Recover slots whose symbol dictionary is damaged

glasstiger
and
claude
committed
c181e4f
Keep the producer alive through a torn symbol dictionary

glasstiger
and
claude
committed
017652f
Assert the new cap-gap config key is honored
glasstiger
committed
dd48d43
Close the untested gaps around the symbol-dict guards

glasstiger
and
claude
committed
521b48d
Set aside the slots recovery cannot rescue, instead of failing

glasstiger
and
claude
committed
aeb92b5
Rebuild the dictionary from the frames still on disk

glasstiger
and
claude
committed
ff53e1f
Pin the producer's recovery seed on the unopenable-dictionary path

glasstiger
and
claude
committed
3b73bc8
Don't key the cap-gap episode off nanoTime's sign

glasstiger
and
claude
committed
98a4ac0

Commits on Jul 15, 2026

Free the recovery mirror on a ctor seed throw

glasstiger
and
claude
committed
63dc52a
Correct stale and stranded symbol-dict comments

glasstiger
and
claude
committed
7299925
Preserve symbol dict on a transient stat fault

glasstiger
and
claude
committed
c0c2ce2
Don't quarantine a fully-acked torn-dict slot

glasstiger
and
claude
committed
0520dc4
Keep foreground cap-gap retries unbounded
glasstiger
committed
8d6a932
Reset cap-gap budget across unrelated outages
glasstiger
committed
84293fa
Prevent orphan adoption during slot quarantine
glasstiger
committed
fe9546f
Verify delivery after torn-slot recovery
glasstiger
committed
6328467
Optimize symbol dictionary recovery and catch-up
glasstiger
committed
5617d79
Harden delta dictionary test correctness
glasstiger
committed
f3282a4
Avoid re-encoding split QWP frames
glasstiger
committed
675d036
Clean up QWP delta dictionary tests
glasstiger
committed
f12c934

Commits on Jul 16, 2026

Commits on Jul 17, 2026

fix: preserve QWP store-and-forward retries
glasstiger
committed
994554e
fix: harden QWP recovery cleanup
glasstiger
committed
20df0eb
Reduce QWP recovery memory and CRC cost
glasstiger
committed
1596dab
Optimize QWP replay and dictionary catch-up
glasstiger
committed
4386ca1
Add QWP edge-case coverage
glasstiger
committed
111f341
Add QWP failure-path coverage
glasstiger
committed
8a1aaf0
Clean up QWP recovery scaffolding
glasstiger
committed
0428fd8
Surface startup endpoint-policy failures

glasstiger
and
claude
committed
9cc550f
Pin the quarantine test on the rename failure

glasstiger
and
claude
committed
68aa31a
Cut symbol-dict overhead and fix two blind guards

glasstiger
and
claude
committed
5ded812
Re-fold recovery at baseline 0 when discarding the dictionary

glasstiger
and
claude
committed
b279eec
Close store-and-forward recovery and drainer gaps

glasstiger
and
claude
committed
ded2edc
Fix connectWithRetry budget overflow

glasstiger
and
claude
committed
16be764
Single-pass symbol-dict append; test window remap

glasstiger
and
claude
committed
042c123
Delete dead recovery symbol-walk methods

glasstiger
and
claude
committed
54e21f8

Commits on Jul 18, 2026

De-reflect cursor tests; de-flake poison timing

glasstiger
and
claude
committed
0a9c3d6
Preserve cap-gap episode; minor test/style nits

glasstiger
and
claude
committed
75d89b8
Fix Windows separator mismatch in SlotLockTest

glasstiger
and
claude
committed
0a3010f

Commits on Jul 19, 2026

Snapshot server batch cap once in sendRow

glasstiger
and
claude
committed
1699c0a
Drop redundant symbol-dict catch-up entry index

glasstiger
and
claude
committed
3314d98
Drop the send loop's unused reconnect budget field

glasstiger
and
claude
committed
2257daa

Commits on Jul 20, 2026

Reclaim logical slot locks on fully-drained close

glasstiger
and
claude
committed
c1587c9
Fix inverted async initial-connect retry docs

glasstiger
and
claude
committed
47b3c9e
Skip a segment whose mmap fault arrives late

glasstiger
and
claude
committed
ece7817
Pin catch-up id tiling and fix stale comments

glasstiger
and
claude
committed
4c10246
Share the slot-lock dir and stop false quarantines

glasstiger
and
claude
committed
54cb19d
Fix Windows path split in orphan-tail drainer test

glasstiger
and
claude
committed
d76ef16
Close four data-integrity holes in the delta symbol dictionary

glasstiger
and
claude
committed
0fe4873
Pin the C3 and C4 fixes with regression tests

glasstiger
and
claude
committed
66bdc0f
Close the remaining four correctness holes in SF recovery

glasstiger
and
claude
committed
3f98352
Cut three redundant walks out of recovery and the flush path

glasstiger
and
claude
committed
509b372
Speed up recovery lookups and the per-symbol varint decode

glasstiger
and
claude
committed
52a9473
Close two test-gate holes; report the third as not reproducible

glasstiger
and
claude
committed
2ee6e8f
Cover three untested-critical rows; explain the two that resist

glasstiger
and
claude
committed
f9bb589
Fix six moderate findings in the SF dictionary paths

glasstiger
and
claude
committed
ba4d458
Close test loops, drop dead walkers, fix misleading comments

glasstiger
and
claude
committed
6d007c7

Commits on Jul 21, 2026

Fix SF recovery, lock and quarantine safety bugs

glasstiger
and
claude
committed
172cfe6

Commits on Jul 28, 2026

Drop the binary-compatibility shims

authored and
Sergei Minaev
committed
db5dc37
Treat a dictionary gap as retriable

authored and
Sergei Minaev
committed
f4f9665
Model the gap rejection in the test decoder

authored and
Sergei Minaev
committed
a225734
Model post-gap connection silence and cap-shrink gaps

authored and
Sergei Minaev
committed
1507e9e
Let an mmap fault reach the dictionary degrade

authored and
Sergei Minaev
committed
11333a4
Bound the catch-up when no cap is advertised

authored and
Sergei Minaev
committed
01288d7
Clear the in-flight holder before rethrowing

authored and
Sergei Minaev
committed
a5b5cc6
Remove the legacy-reader barrier

authored and
Sergei Minaev
committed
19f7536
Make a shared sf_dir opt-in

authored and
Sergei Minaev
committed
b795b4b
Add end-to-end permission test for sf_dir_shared

authored and
Sergei Minaev
committed
6ca6ad2
Finish close() when the last batch is rejected

authored and
Sergei Minaev
committed
4d245ff
Refuse a fresh slot with an unclearable dictionary

authored and
Sergei Minaev
committed
3fe1828
Cover the dictionary refusal through a real EISDIR

authored and
Sergei Minaev
committed
627507c
Separate the words in the cap-gap config key

authored and
Sergei Minaev
committed
7494fee
Fail closed when recovery skips a segment

authored and
Sergei Minaev
committed
25d253f
Add the missing all-skipped regression test

authored and
Sergei Minaev
committed
6018642
Quarantine a slot whose construction itself refuses

authored and
Sergei Minaev
committed
af513a9
Refuse a skip before the gap it can open, not after

authored and
Sergei Minaev
committed
ab974c4
Carry a generation id in the segment header

authored and
Sergei Minaev
committed
9fa9067

Commits on Jul 29, 2026

Guard rotation against a closing ring; test generation propagation

authored and
Sergei Minaev
committed
84fa7cd
Correct comments describing removed server behaviour

authored and
Sergei Minaev
committed
8c05107
Fix sticky-bit arithmetic in SfDirPermissionsTest javadoc

authored and
Sergei Minaev
committed
04f67e8
Quarantine unreplayable slots; fix test gaps

authored and
Sergei Minaev
committed
0b5a2c4
Merge origin/main into qwp-delta-symbol-dict

Sergei Minaev
authored and
Sergei Minaev
committed
8813206
fix(sf): refuse a fresh start when a drained-slot leftover survives removal

Sergei Minaev
authored and
Sergei Minaev
committed
45ec2fa
fix(sf): treat a drained-slot leftover unlink failure as operational, not terminal

Sergei Minaev
authored and
Sergei Minaev
committed
c597421
test(sf): narrow the leftover-refusal assertion to its own message

Sergei Minaev
authored and
Sergei Minaev
committed
7e327b3

Commits on Jul 30, 2026

refactor(sf): drop the lineage stamp from the persisted symbol dictionary

authored and
Sergei Minaev
committed
0959b46
test(sf): pin the HEADER_SIZE routing boundary in PersistedSymbolDict.open

authored and
Sergei Minaev
committed
0df1335
test(sf): make the HEADER_SIZE boundary test discriminate load from recreate

authored and
Sergei Minaev
committed
5fadc42
refactor(sf): drop the recovered-chain lineage agreement check

Sergei Minaev
authored and
Sergei Minaev
committed
b5728a5
docs(sf): fix UnreplayableSlotException javadoc to match live throw sites

Sergei Minaev
authored and
Sergei Minaev
committed
d3bff09
refactor(sf): shrink the segment header back to 24 bytes, version 1

Sergei Minaev
authored and
Sergei Minaev
committed
78cd3b6
fix(sf): update stale header-size comments, sharpen a rename-failure

authored and
Sergei Minaev
committed
fd2d0f1

Commits on Jul 31, 2026

Remove the sf_dir_shared opt-in

Sergei Minaev
authored and
Sergei Minaev
committed
724fee8
Unlink the pid sidecar before the lock file

Sergei Minaev
authored and
Sergei Minaev
committed
27b5276
Latch hasEverConnected after the catch-up succeeds

Sergei Minaev
authored and
Sergei Minaev
committed
3383f29
fix(qwp): refuse symbol registration past the server dictionary cap

authored and
Sergei Minaev
committed
8f8ecf9
test(qwp): pin the row-API contract at the symbol dictionary cap

authored and
Sergei Minaev
committed
68960c3
Fix SegmentSkipQuarantineTest's racy rotation precondition

authored and
Sergei Minaev
committed
88568f7
Fix alphabetical ordering and javadoc wrap in QwpConstants

authored and
Sergei Minaev
committed
164bb9b
Harden the build()-time quarantine path

authored and
Sergei Minaev
committed
3c7dcc9
Pin the quarantine SenderError dispatch

authored and
Sergei Minaev
committed
4d8702c
Fold sendRow's second column walk into nextRow

authored and
Sergei Minaev
committed
0d50275
Name reset() in the batch-too-large recovery hint

authored and
Sergei Minaev
committed
0913496

Commits on Aug 1, 2026

Create test dictionaries through openClean, not open

authored and
Sergei Minaev
committed
df199dd
Fail loudly on transient symbol-dict recovery faults

authored and
Sergei Minaev
committed
a16d182
Align recovery fixtures with the no-fabrication contract

authored and
Sergei Minaev
committed
f5faf1c
Pin the fail-loud break of the misattribution chain

authored and
Sergei Minaev
committed
425eedc
Route dict existence checks on stat errno

authored and
Sergei Minaev
committed
6193d10
Expose durable side-file bytes from the symbol dictionary

authored and
Sergei Minaev
committed
141515e
Count side-file bytes against the SF disk cap

authored and
Sergei Minaev
committed
4af3691
Fix disk-full warn to not promise a remedy trim can't deliver

authored and
Sergei Minaev
committed
091476a
Wire the dict side-file gauge into manager registration

authored and
Sergei Minaev
committed
e1ac106
Harden the cap control test and make the dict gauge wait-free

authored and
Sergei Minaev
committed
a8e8752
Rebuild the torn-slot fixture through the manager trim and re-enable the fully-acked resume test

authored and
Sergei Minaev
committed
8be7670
Re-enable the torn-dict guard test on the manager-path fixture

authored and
Sergei Minaev
committed
8328f21
Add an end-to-end DICTIONARY_GAP NACK recycle-and-replay test

authored and
Sergei Minaev
committed
71856dc
Cover both branches of the close() drain-timeout outage naming

authored and
Sergei Minaev
committed
b1d5524
Polish test comments, license banner, and explicit sync-connect config

authored and
Sergei Minaev
committed
0b363eb

Commits on Aug 3, 2026

Clamp the OK-path ACK wire sequence at zero

authored and
Sergei Minaev
committed
8484207
Add DATA_LOSS category for abandoned SF data

authored and
Sergei Minaev
committed
850dff7
Reclassify slot quarantine as DATA_LOSS

authored and
Sergei Minaev
committed
69b3798
Notify the pool errorHandler of recovery data loss

authored and
Sergei Minaev
committed
9a17e6a
Give the recovery-suppression test a real event to suppress

authored and
Sergei Minaev
committed
16a8784
Report drainer slot abandonment as DATA_LOSS

authored and
Sergei Minaev
committed
fbff733
Adopt DATA_LOSS at the remaining drainer sites

authored and
Sergei Minaev
committed
dd483d1
Polish DATA_LOSS logging and drainer dispatch edges

authored and
Sergei Minaev
committed
239c4e4
Cover the mid-life-degraded slot shape

glasstiger
and
claude
committed
73f09da
Merge branch 'qwp-delta-symbol-dict' of https://github.com/questdb/java-questdb-client into qwp-delta-symbol-dict
glasstiger
committed
c249894
Merge remote-tracking branch 'origin/main' into qwp-delta-symbol-dict
glasstiger
committed
b8a4e46
Cover four untested error/recovery paths

glasstiger
and
claude
committed
ec10320
Fold the row-size guard into nextRow's walk

authored and
Sergei Minaev
committed
0b47d85
Keep in-progress state until the row commits

authored and
Sergei Minaev
committed
f5d8bc1
Chunk the full dictionary into its own frames

glasstiger
and
claude
committed
8092cfe
Pin three fixes that no test could catch

glasstiger
and
claude
committed
cbc3fec
Read errno beside its stat, and fix four wrong comments

glasstiger
and
claude
committed
967cac0
Merge branch 'qwp-delta-symbol-dict' of https://github.com/questdb/java-questdb-client into qwp-delta-symbol-dict
glasstiger
committed
6ce153e

Commits on Aug 4, 2026

Chunk the dictionary when no body fits beside it

authored and
Sergei Minaev
committed
3d48519
Fold the PR-review red tests into SegmentRingTest

glasstiger
and
claude
committed
2b131a4
Pin the fallback's delta-mode gate

jovfer
and
claude
committed
36195cc
Pin the close-time drain of earlier rows

glasstiger
and
claude
committed
3ef580a
Merge branch 'qwp-delta-symbol-dict' of https://github.com/questdb/java-questdb-client into qwp-delta-symbol-dict
glasstiger
committed
ccef013
Keep the sender usable after an over-cap batch

glasstiger
and
claude
committed
ddab88d
Keep the SF disk cap from wedging ingestion

glasstiger
and
claude
committed
eb32488
Chunk the dictionary only when the batch can ship

glasstiger
and
claude
committed
d3219c5
Return never-shipped symbol ids on reset()

glasstiger
and
claude
committed
7238e19
Stop auto-flush from moving a test's throw site

glasstiger
and
claude
committed
8f5ed4f

Commits on Aug 5, 2026

Stop an unusable symbol dictionary wedging a slot

glasstiger
and
claude
committed
8615c55
Commit dictionary chunks a failed flush stranded

glasstiger
and
claude
committed
8ae795a