Wire-level CW=8 negotiation + D8PSK gate SNR>=22 (throughput push)#27
Merged
Wire-level CW=8 negotiation + D8PSK gate SNR>=22 (throughput push)#27
Conversation
D8PSK was disabled in the OFDM rate ladder with a "TEMPORARY: fails
on any fading" note that's been stale since the 2026-03-15 CPE
correction + per-symbol pilot tracking landed. cli_simulator sweeps
on the experimental branch confirm D8PSK now works in fading:
good fading (cli_simulator 7-message test):
R1/2 SNR=8: FAIL (cliff)
R1/2 SNR=10: PASS, 4 retx
R1/2 SNR=12: PASS, 2 retx
R1/2 SNR=15: PASS, 0 retx
R2/3 SNR=10: PASS, 28 retx (high — below cliff)
R2/3 SNR=12: PASS, 45 retx (very high)
R2/3 SNR=15: PASS, 0 retx
R2/3 SNR=20: PASS, 1 retx
R3/4 SNR=20: PASS, 6 retx (border, AWGN-only)
moderate fading: D8PSK R1/2 stable at SNR>=15 (3-6 retx).
Conservative gate added in waveform_selection.hpp:
- D8PSK R3/4 at AWGN (fading<0.15) AND SNR>=22
- D8PSK R2/3 at AWGN AND SNR>=18, OR good fading (fading<0.65) AND SNR>=15
- D8PSK R1/2 at good fading (fading<0.65) AND SNR>=10
- DQPSK fallback otherwise (preserves all documented baselines)
Throughput math at the most common operating point:
Before: DQPSK R1/2 good fading SNR=15 → ~2.3 kbps usable
After: D8PSK R1/2 good fading SNR=15 → ~3.4 kbps usable (+47%)
After: D8PSK R2/3 good fading SNR=15 → ~5.0 kbps usable (+117%)
(adaptive promotion past bootstrap cap; first MODE_CHANGE)
In auto-rate cli_simulator regression sweep, all conditions PASS:
good fading SNR=12: D8PSK R1/2, 1 retx, PASS
good fading SNR=15: D8PSK R1/2, 0 retx, PASS
good fading SNR=20: D8PSK R1/2, 0 retx, PASS (R2/3 after MODE_CHANGE)
good fading SNR=25: D8PSK R1/2, 0 retx, PASS
moderate SNR=12-25: DQPSK R1/4 to R1/2 unchanged, all PASS
Tests updated: test_protocol mode_change test (was DQPSK R2/3 at
SNR=22, now D8PSK R2/3 capped); test_waveform_policy expectations
flipped from DQPSK to D8PSK at the new gate thresholds, plus 4
new boundary checks pinning the D8PSK ladder.
ctest 35/35.
Branch: experimental/throughput-push (NOT main). Codex audited
this change as the highest-leverage tonight lever among 20
candidates surveyed; remaining levers (16-QAM, 32-QAM, larger
LDPC, HARQ-IR, per-subcarrier bit loading) all need multi-day
research-level work and are deferred.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Codex (audit, 2 passes)
7-message sweep suggested D8PSK R2/3 at SNR>=15 good fading was
zero-retx. Follow-up 5KB file-transfer test at SNR=18 good fading
showed 31 retx for 28 frames — D8PSK R2/3 is too tight there. The
short test doesn't expose the failure mode the longer transfer does
because a 7-message exchange completes before the channel hits its
worst fade events.
Tightened the D8PSK R2/3 gate to three conditional tiers:
- Clean AWGN (fading<0.10) at SNR>=15: R2/3 (preserves the
adaptive-upgrade test path; sweep clean)
- Slight residual fading (<0.15) at SNR>=18: R2/3
- Real good fading (<0.65) at SNR>=20: R2/3 (file-transfer-tested)
D8PSK R1/2 stays at SNR>=10 fading<0.65 (unchanged) — that's the
dependable +47 % win over DQPSK R1/2 that doesn't break under
file-transfer load.
Verified:
ctest 35/35 (test_connection_adaptive R2/3 promotion path stays
reachable at SNR=15 fading=0.05)
cli_simulator --snr 18 --fading good --file 5120 --test: PASS
(was FAIL with the looser R2/3 gate)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
isHighThroughputOFDMMode() previously rejected anything that wasn't DQPSK, so the new D8PSK R1/2+ rates fell back to the default OFDM window=8. Extending the predicate to D8PSK R1/2/R2/3/R3/4 puts the new rates on the same window=16 selective-repeat track that DQPSK high-rate modes already use. isSpeculativeHighRateOFDM() also now recognizes D8PSK R2/3 and R3/4 as speculative — so window=16 applies only on near-AWGN, falling back to window=8 in real fading. R1/2 (the dependable D8PSK win) is non-speculative and gets window=16 unconditionally when the D8PSK gate fires (SNR>=10 fading<0.65). Verified: - ctest 35/35 (test_connection_policy padding-policy assertion flipped: D8PSK R2/3 now fires the same partial-burst padding policy as DQPSK R2/3, which is the right behavior post-promotion) - cli_simulator 5KB file transfer SNR=18 good fading auto-rate: PASS, window=16 D8PSK R1/2 in initial mode log - cli_simulator SNR=15 good fading R1/4 documented baseline: PASS Combined throughput estimate on the experimental branch vs main: Good fading SNR=15: ~2.3 kbps → ~3.4 kbps (D8PSK R1/2 + window=16) Good fading SNR=20: ~3.4 kbps → ~5.0 kbps (D8PSK R2/3 + window=16) AWGN SNR=27: ~3.9 kbps → ~5.9 kbps (D8PSK R3/4 + window=16) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Captures the experimental branch's audit + execution + measurement. Bottom line: 10 kbps in good fading is unreachable per Codex Shannon- ceiling math. Real wins delivered: - D8PSK R1/2 in good fading SNR>=10: +47 % over DQPSK R1/2 - D8PSK R2/3 in clean conditions: +47 % over DQPSK R2/3 - D8PSK R3/4 on AWGN SNR>=24: +47 % over DQPSK R3/4 - ARQ window=16 extended to D8PSK rates Combined: ~3.4 kbps in typical good fading SNR=15 (was ~2.3 kbps), ~5.0 kbps in clean conditions SNR=20 (was ~3.4 kbps). Documents three Codex audit passes, the sweep that picked the gate thresholds, and the file-transfer stress test that tightened R2/3. Lists the deferred levers (16-QAM, larger LDPC, HARQ-IR, per- subcarrier bit loading) for the next throughput attempt. Branch: experimental/throughput-push (not main). Awaits OTA validation before merging. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: Codex (audit, 3 passes)
Mac↔Pi5 audio loopback A/B with synthetic-channel injection (5KB
file transfers, R1/2 forced, good fading) revealed a 10 dB sim-vs-
hardware gap on D8PSK:
Forced D8PSK vs DQPSK R1/2 at SNR=15-20 good fading:
SNR=15: DQPSK 1078 bps (2 retx) > D8PSK 728 bps (5-40 retx)
SNR=18: DQPSK 1234 bps (0 retx) > D8PSK 641 bps (38 retx)
SNR=20: DQPSK 1247 bps (0 retx) < D8PSK 1595 bps (0 retx) ← +28 % win
Watterson simulator showed D8PSK R1/2 working cleanly at SNR=10
because Watterson doesn't model: soundcard ADC/DAC quantization,
audio AGC residual, anti-aliasing filter response, real audio-chain
phase noise. D8PSK's 8-phase decision is far more sensitive to
those than DQPSK's 4-phase.
Gates tightened based on real measurements:
D8PSK R1/2: was SNR>=10 fading<0.65; now SNR>=20 fading<0.65
D8PSK R2/3: was SNR>=15-20 fading<0.65 (multi-tier); now AWGN
only — fading<0.10 SNR>=18 OR fading<0.15 SNR>=22
D8PSK R3/4: unchanged — AWGN-only SNR>=24
DQPSK fallback for everything below the new D8PSK floor
Adaptive promotion test: SNR=20 auto-rate D8PSK R1/2 at start;
adaptive only attempts R2/3 when fading drops below 0.10 (the
post-promotion measurement window) so the 486-bps "promotion-
collapsed" failure path observed earlier no longer triggers.
Also added --mod option to tools/run_hw_test.sh so future hardware
A/B comparisons can force modulation directly.
Verified:
- ctest 35/35
- Auto-rate SNR=15 good fading 5KB: PASS, 904 bps (DQPSK R1/2)
- Auto-rate SNR=20 good fading 5KB: PASS, 1130 bps (D8PSK R1/2)
- Hardware smoke 4/4: PASS
Honest position on the original "10 kbps in good fading" ask:
- Theoretical sim ceiling: ~5 kbps with D8PSK R2/3 + window=16
- Real hardware ceiling: ~1.6 kbps payload (D8PSK R1/2 SNR=20 forced)
- 10 kbps is physically unreachable on this PHY shape; this branch
delivers a real ~28 % hardware-measured win at SNR>=20 good
fading, no regression on documented baselines.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5KB file-transfer A/B on Mac↔Pi5 audio loopback at SNR=15/18/20 good fading showed the simulator overpromised D8PSK by 10 dB. Real cliff is SNR=20 in fading; below that, DQPSK is faster than D8PSK because the 8-phase modulation eats more noise from soundcard quantization + audio-chain phase jitter than the LDPC code can recover. Hardware-validated D8PSK R1/2 win at SNR>=20 good fading: +28 % (1247 → 1595 bps payload, both 0 retx). 10 kbps remains unreachable. On hardware the practical ceiling is ~1.6 kbps payload at the operating points we care about. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User pushed back on apparent throughput regression. Investigation
findings, all from real Mac↔Pi5 audio loopback:
(a) "We had 2000+ bps DQPSK" was correct — for R2/3 and R3/4, not
R1/2 (the fading-fallback rate).
DQPSK R2/3 good SNR=20: 1422 bps (sim said 1837)
DQPSK R3/4 AWGN SNR=25: 2058 bps (sim said 2508)
(b) "Window 16 doesn't work on fading" — not borne out by hardware
A/B at SNR=15 good/moderate. Window=16 actually wins by 12-28 %
over window=8 across every tested condition. The 8-frame burst
interleaver still aligns: window=16 = 2 burst groups in series.
Window=16 restored.
(c) D8PSK R2/3 on AWGN SNR=22+ is the real win discovered tonight:
2382-2410 bps payload, 0 retx, beats DQPSK R3/4 (2058) at the
same channel. Auto-rate triggers it correctly at SNR=22 AWGN
delivering 2406 bps with 0 retx.
(d) D8PSK R3/4 ceiling on hardware: ~2620 bps at SNR=30 AWGN —
diminishing returns vs R2/3.
Updated THROUGHPUT_PUSH_2026-05-04.md with these measurements and
the auto-rate validation. Honest 10 kbps answer remains: no, but
the 2 kbps target is achievable today on D8PSK R2/3 in AWGN.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User pushed: "I feel main has more speed." Settled with data — rebuilt main on Pi5, ran identical conditions to the experimental- branch tests: Test MAIN EXPERIMENTAL Δ DQPSK R1/2 SNR=15 good (forced) 1073 bps 1077 bps tied DQPSK R2/3 SNR=20 good (forced) 1415 bps 1422 bps tied DQPSK R3/4 SNR=25 AWGN (forced) 2057 bps 2058 bps tied AUTO SNR=22 AWGN 1837 bps 2406 bps +31 % Forced-mode tests are tied — the experimental commits don't change the modulator or LDPC layer, only the rate-ladder gates. Experimental wins where the gate fires (auto-rate clean conditions): main picks DQPSK R2/3, experimental picks D8PSK R2/3, same code rate but 1.5× bits-per-symbol = +31 % real throughput, 0 retx either branch. No regression on any tested point. Main is not faster than this branch on this hardware. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User pushed: "1077 bps DQPSK R1/2 SNR=15 good is too low — investigate." Hardware sweep on Mac↔Pi5 with --cw-count 8 (existing CLI opt-in from commit 6cc77ea): Mode/Rate Channel SNR cw=4 cw=8 Δ DQPSK R1/2 good 15 1077 bps 1615 bps +50 % DQPSK R1/2 good 12 ~1100 bps 1594 bps +45 % DQPSK R1/2 moderate 15 1234 bps 1594 bps +29 % DQPSK R3/4 AWGN 25 2057 bps 2360 bps +14 % D8PSK R2/3 AWGN 22 2406 bps 2906 bps +21 % D8PSK R3/4 AWGN 27 2620 bps 3127 bps +19 % CW=8 wins everywhere except R2/3 in fading (14 retx vs 0 with cw=4) because longer frames hit fade events more often. R1/2 — the dominant fading operating mode — gets a clean +50 % at SNR=15 with zero retx penalty. The math: 5.3 s SACK-deferral per ARQ window is fixed overhead. CW=8 doubles payload-per-frame, so it amortizes that fixed cost across twice the bytes. The lever has been in --cw-count since 6cc77ea; it just isn't on by default. Initially attempted to make this auto rate-aware in Connection::configureArqForCurrentDataMode (R1/2 → CW=8 in the connection setup) but it tripped the test_connection_adaptive clean-window accumulator's timing model (longer frames need more ticks for 3 clean windows). Reverted the auto-bump; documented as a CLI opt-in. The auto-promote path needs a follow-up to make adaptive window-counting CW-aware before it can ship as default. For the user's stated goal of "more than 1077 bps DQPSK R1/2 SNR=15 good fading": deliverable today via `--cw-count 8` = 1615 bps. Absolute hardware ceiling on this rig is ~3.1 kbps (D8PSK R3/4 AWGN SNR=27 + cw=8); 10 kbps remains unreachable. ctest 35/35. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Attempted to ship CW=8 as the auto-default for R1/2 / R3/4 / R2/3-
AWGN via recommendCWCount() helper in connection_policy.hpp +
auto-application in Connection::configureArqForCurrentDataMode().
Hit two blockers:
1. Encoder/decoder don't see protocol-side CW changes. When protocol
bumped CW=4→8 on connect, StreamingEncoder / StreamingDecoder
still had CW=4. Mid-handshake mismatch caused
`libc++abi: terminating due to uncaught exception of type
std::__1::system_error: mutex lock failed: Invalid argument`
during the first burst-flush on hardware (Mac↔Pi5). The modem
layer is set up upstream of Connection by ProtocolEngine /
cli_simulator before the connection knows the data rate, with
no callback path for CW changes downstream.
2. Global kDefaultFixedFrameCodewords = 4 → 8 as a fallback broke
3 ctest suites (ConnectionPolicy, ConnectionAdaptive, FrameV2
subprocess aborted). The constant feeds into frame-format math,
fixed-buffer allocations, and adaptive timing models that all
assume CW=4 baseline.
Both attempts reverted. Branch experimental/throughput-push stays at
9008011 documenting CW=8 as a CLI-opt-in win (--cw-count 8 today
gives the +50 % real-hardware throughput on DQPSK R1/2 SNR=15 good
fading).
Documented the refactor that ships CW=8 as default: a callback
path for protocol→modem CW-count change notifications + adaptive-
timing-test cw-aware tick budget. ~1-2 days of careful work.
ctest 35/35 (post-revert).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Earlier attempt set CW from a host-side data-mode-changed callback that re-entered ProtocolEngine::mutex_; non-recursive std::mutex deadlocked the responder, CONNECT_ACK never drained from tx_queue_, initiator timed out. Caught with seed=1 sim A/B (baseline 10.5s handshake, patched 120s timeout, 100% reproducible). Codex (gpt-5.5 xhigh) review surfaced three more hazards beyond the deadlock: stale CONNECT_ACK retry timer, decoder fallback to configured fixed_frame_codewords_ when header read fails, and "callbacks fire under the protocol mutex". Bottom line: do not ship "both sides recompute" as the agreement mechanism — carry CW explicitly on the wire. Wire: - ConnectFrame.PAYLOAD_SIZE 25→26B, new data_frame_cw_count byte. CONNECT carries initiator's forced CW (0=AUTO); CONNECT_ACK carries responder's chosen value. Initiator applies the echoed value. - ControlFrame::ModeChangeInfo gains data_frame_cw_count via payload[5] (was reserved — no size change). Policy: - recommendCWCount(rate) is rate-only: R1/2, R2/3, R3/4 → 8; R1/4 → 4. No SNR/fading dependency, so cross-peer agreement collapses to "both peers ran the same rate negotiation". - applyDataMode(mod, rate, cw_count=0): explicit cw from MODE_CHANGE, else auto via recommendCWCount(rate). Triggers requeuePendingChunks on rate-changed OR cw-changed (was rate-changed only). - DataModeChangedCallback signature now (mod, rate, cw_count, snr, fading). Hosts poke encoder+decoder directly from the param — no protocol_.setForcedFrameCodewords() inside the callback, no mutex re-entry. - CONNECT_ACK retry timer is now computed AFTER cw is finalized. CLI override: - setForcedFrameCodewords(cw, forced=true). forced=true marks config_.forced_cw_count for one-sided wire propagation; forced=false is the boot-time path (host wiring up encoder/decoder) that does NOT mark forced and thus does not bypass the responder's auto-pick. - cli_simulator tracks cw_count_forced_ explicitly so only --cw-count trips the forced path; default init goes through forced=false. Sim verification (seed=1, SNR=15 good fading, 5KB DQPSK R1/2 auto): baseline (CW=4 default): handshake at 10.5s, transfer done by 39s patched: both peers "Negotiated CW count: 8", transfer done by 36s Hardware A/B (Mac↔Pi5 audio loopback, --inject good fading SNR=15, DQPSK R1/2 5KB): run 1 (auto, init bug had forced=true): 1233 bps, 39 frames, 0 retx run 2 (auto, after init bug fixed): 1448 bps, 19 frames, 0 retx In-session +17%; frames halved (39→19) confirms CW=8 in effect. Prior force-CW measurements: 1077 bps (CW=4) → 1615 bps (CW=8). ctest: 35/35 green incl. ConnectionPolicy, ConnectionAdaptive, and FrameV2 — the suites that had broken on the previous abandoned attempt. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codex review caught: requestModeChange() always set pending_cw_count_ = recommendCWCount(new_rate), so an operator-forced --cw-count 4 was silently lost on the first mid-transfer MODE_CHANGE (rate adapted up → CW jumped to 8 against the operator's wish). The config_.forced_cw_count field already exists for exactly this purpose at CONNECT time; just check it again at MODE_CHANGE time. Cast wrapper kept compatible with int return from sanitize + recommendCWCount. Also adds 2026-05-04 CHANGELOG entry covering the full wire-negotiation work (commits 1a98b4d + this one). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
10-seed hardware sweep (Mac↔Pi5 audio loopback, 5KB DQPSK auto,
good fading injected, 2026-05-04 post-CW=8 wire negotiation):
SNR=20 good: D8PSK retx-hit 38 % (3/8 storms incl. 270 bps)
mean 1448 bps ≈ DQPSK alt 1444 bps — wash with tail.
SNR=22 good: D8PSK retx-hit 17 % (1/6 single retx, no storms)
mean 1783 bps vs DQPSK 1450 — +23 % real win.
SNR=24 good: D8PSK retx-hit 43 % (3/7 incl. 2 FAILs at 320-374 bps,
17-78 retx). Counterintuitively WORSE than 22.
The single-seed CLAUDE.md datapoint (SNR=20 D8PSK 1595 bps clean)
that motivated the prior SNR>=20 gate was unrepresentative of the
distribution. Variance hidden in single-seed measurements.
Storms aren't predictable from bulk fading_index (SNR=22 storm hit
at fading=0.45; SNR=24 storms hit at 0.52/0.53/0.58 — all median),
so tightening fading further doesn't help. SNR=22 is the floor.
Test updated to assert the new ladder: SNR=22 good promotes to
D8PSK; SNR=20 good stays DQPSK.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CONNECT_ACK.data_frame_cw_countbyte (frame payload 25→26 B) andMODE_CHANGE.payload[5](was reserved). Initiator embeds forced CW (or 0=AUTO), responder picks viarecommendCWCount(rate)and echoes the chosen value.--cw-count Nis now a one-sided forced override that propagates over the wire.What's broken / what's fixed
recommendCWCount(rate)is rate-only — both peers compute the same CW from the same negotiated rate, no SNR/fading drift hazard. R1/2, R2/3, R3/4 → 8; R1/4 → 4.applyDataMode(mod, rate, cw_count=0)triggersrequeuePendingChunkson rate-changed OR cw-changed (was rate-changed only).requestModeChangehonorsconfig_.forced_cw_countso--cw-countsurvives mid-transfer rate adaptation (Codex finding 1, fixed in 2ff2332).DataModeChangedCallbacksignature:(mod, rate, cw_count, snr, fading)— host code uses param directly instead of re-enteringprotocol_.setForcedFrameCodewords()(the deadlock fix).Test plan
ctest --test-dir build -j4— 35/35 green incl.ConnectionPolicy,ConnectionAdaptive,FrameV2,WaveformPolicy--cw-count 4forced override sanity check after mergeHardware data motivating the throughput case
🤖 Generated with Claude Code