Skip to content

fakeserver: CompressShape shapes the COMPRESS DEFLATE output stream - #15

Merged
go-while merged 2 commits into
mainfrom
compress-shaping
Nov 12, 2026
Merged

go-while merged 2 commits into
mainfrom
compress-shaping

Conversation

@go-while

@go-while go-while commented Nov 12, 2026 •

Copy link
Copy Markdown
Owner

Shapes the fake server's COMPRESS DEFLATE (RFC 8054) output stream for client tests (survey item B6; go-pugleaf asked for split writes and the empty flush with the 206, partial flush as nice-to-have).

API

CompressShape (new field CompressShape on StoreOptions, PeerOptions and Step; Conn.StartCompressShaped(shape) for a Handler that writes its own 206):

field meaning
Flush FlushSync (default); FlushPartial: zlib's Z_PARTIAL_FLUSH like INN's nnrpd (empty fixed-Huffman block, no 00 00 FF FF, the last 0-7 bits unsent); FlushFinish: the first write after the 206 ends the stream (final block), later writes are plain
Split the existing Split type applied to the compressed bytes of each write: At (a negative offset counts from the end, so -1..-3 cut inside the sync marker), Size, Gap (bounded by the Deadline, via the conn's Sleep)
EmptyFlushWith206 the 206 line and an empty flush block in one socket write
With206 []byte deflated and flushed into the 206's socket write (compressed payload pipelined behind the 206, what TestCliCompressPipelinedAfter206 hand-builds)
AfterEnd []byte with FlushFinish: bytes after the final block, in the same write

Behaviour changes

None: the zero value takes the old path (one default-level flate writer, sync flush, one socket write per write).

Design

  • Partial flush: compress/flate cannot emit it (its marker is a byte-aligned stored block), so FlushPartial uses a ~40-line bit writer with fixed-Huffman literal blocks (no matches; output is larger, correctness is what matters). zlib >= 1.2.5.2 semantics: data block, then one empty static block (3 header bits + 7-bit EOB), only whole bytes sent.
  • compress/flate's own reader holds data until a stored block, so it cannot read a partial-flush stream promptly (go-pugleaf's test says the same); the tests read it with the kit's internal/inflate, and one unit test completes the stream (pad + final stored block) to check it with compress/flate's reader, including all 256 literal values.
  • The 206 and its tail are one Write in writeKind (the encoder is created there and handed to startCompress, so bit state continues). That works in the kit's handlers only; a custom Handler that writes its own 206 cannot use EmptyFlushWith206 / With206, documented.
  • Pieces of a Split are counted per write (each reply), not over the stream.

Skipped

  • Z_FINISH with later compressed writes (a new stream): after the final block the server writes plain bytes; that covers data-after-stream-end and stream-finished-exactly.
  • zlib's suppression of a repeated flush with no new input: every write emits its flush block.
  • The older zlib's second empty block (< 1.2.5.2).

Tests

New fakeserver/compressshape_test.go: default stays one write per reply; exact socket pieces (recorded writes plus readPieces with a lower-bound timing check, never read boundaries); inside-the-marker cuts; byte-wise delivery to compress/flate's reader; empty flush with the 206 (sync and partial, recorded as one write, nothing after the 206 without the option); With206; partial flush (prompt delivery, no marker, encoder vs compress/flate); finish with and without trailing bytes; Peer and Script step; StartCompressShaped.

go vet ./..., go test -race ./..., go test -race -count=10 -run Compress ./fakeserver green.

🤖 Generated with Claude Code

go-while and others added 2 commits November 12, 2026 13:43
Flush kind (sync, zlib-style partial flush like INN's nnrpd, final block with
trailing bytes), compressed bytes split across socket writes (offsets from the
start or end, so also inside the 00 00 FF FF marker), an empty flush or pipelined
payload in the same socket write as the 206. Option on StoreOptions, PeerOptions,
Step and Conn.StartCompressShaped; the zero value is the old behaviour.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Arm the 206 tail only when compression will start, keep the caller's write
deadline across a split gap, record writes before they happen in the test
recorder, terminate the partial-flush test stream correctly, document the
unsent bits and the tail in fault offsets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@go-while
go-while merged commit 11c2946 into main Nov 12, 2026
8 checks passed
@go-while
go-while deleted the compress-shaping branch November 12, 2026 14:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant