Skip to content

Add an optional micro-flac backend for the FLAC decoder - #185

Merged
rtissera merged 4 commits into
masterfrom
microflac-opt-in
Sep 10, 2026
Merged

Add an optional micro-flac backend for the FLAC decoder#185
rtissera merged 4 commits into
masterfrom
microflac-opt-in

Conversation

@rtissera

@rtissera rtissera commented Sep 6, 2026

Copy link
Copy Markdown
Owner

CHDR_FLAC_BACKEND=microflac decodes FLAC through micro-flac instead of dr_flac, behind the same flac_decoder interface. Exactly one backend is compiled. dr_flac stays the default and nothing changes for anyone who does not ask for this.

micro-flac is Apache-2.0 and C++, so it is fetched at build time from a pinned commit rather than vendored: this tree stays BSD-3 for its other consumers, and a default build neither touches the network nor needs a C++ compiler. CHDR_MICROFLAC_SOURCE_DIR points at a local checkout to build offline.

Measured on an ESP32-S3, every hunk of five images from the card

image dr_flac micro-flac
disc A (cdzs 92.9%) 231617 ms 230054 ms 1.007x
disc M (cdlz 88.6%) 334679 ms 338756 ms 0.988x
disc B (cdfl 79%) 183136 ms 158877 ms 1.153x
disc C (cdfl 64.6%) 150508 ms 127735 ms 1.178x
image N (non-CD) 59764 ms 59829 ms 0.999x
total 959703 ms 915250 ms 1.049x

15-18% where FLAC carries the image, neutral where it does not, and one file 1.2% slower. That one is not I/O — the card costs 0.875 ms/hunk on both sides, identically — it is 0.060 ms/hunk of decode CPU, on an image where cdfl is a minority codec. It is not explained, and it is stated rather than averaged away.

It uses less memory, which was not true of the first attempt

At the same point of the same image: 198 KB of heap in use against dr_flac's 210, and a largest free block of 52 KB against 39. Over the whole run the smallest largest-free-block is 36 KB against dr_flac's 31 — that figure, not the peak, is what decides whether a later allocation of that size succeeds.

That came from one field. micro-flac checks the caller's output buffer against a whole block on every call, sized from the STREAMINFO block-size field — and the header libchdr synthesises carries block_size * num_channels where the format specifies inter-channel samples. It was therefore demanding 18816 bytes at a point where only 10176 remained, which forced every hunk through an intermediate buffer. Writing the specified value halves the demand to 9408, and the caller's hunk is then large enough for every block of it.

Neither value changes the decoded bytes — each frame carries its own block size, and that is what both decoders read. A buffer smaller than one block, or a final block that would overrun it, still falls back to a retained scratch; for CD geometry it is never allocated.

Before this, the backend cost 218 KB and left 33 KB — worse than dr_flac. The three configurations are worth comparing directly, because they separate two effects that look like one: removing the copy is what fixed the memory and the small regression on non-FLAC images, while the 15-18% on cdfl images comes from the decoder itself and is unchanged by it.

One incidental correctness gain

dr_flac writes int16_t straight into the caller's buffer, which is undefined on an odd pointer and is why chd_read documents a 2-byte alignment requirement. Under UBSan with a deliberately misaligned buffer, dr_flac aborts on all ten odd-aligned cases and micro-flac completes every one, with no sanitizer diagnostic at all.

Verification

Byte-identical to the dr_flac backend throughout:

  • the 17 seeds in three read orders
  • both settings of CHDR_CD_SCRATCH_BUFFER — the branch this replaces predated the in-place CD spread and had never been run against it
  • 4253 real images, 400 hunks sampled each
  • 255 ASan/UBSan runs across seed x access pattern x alignment, no diagnostic on this backend
  • built for ESP32-P4 and ESP32-S3, and run on an S3

The new workflow builds both backends in both scratch settings and diffs the seeds on every change — an opt-in C++ path fetched at a pinned SHA that nothing compiles would rot otherwise. It also configures a default build with a compiler path that does not exist, so enable_language(CXX) cannot silently escape the backend's own branch.

Not in this PR

The same header field costs dr_flac 18816 bytes of peak heap for the same reason, and correcting it there is byte-identical too. That is a separate change with its own blast radius — it affects every build, not an opt-in one — so it gets its own PR.


Re-measured after #188

The numbers above predate #188, which fixed the synthesised STREAMINFO block size and took 18816 bytes off dr_flac. That changed the comparison, so everything here was re-measured on the merged tree.

ESP32-S3, 11 real discs off a card, five codec profiles, BENCH_HUNK_CAP=1500:

profile hunkbytes dr_flac micro-flac ratio
cdlz cdzl cdfl 19584 5630 4640 1.213
cdlz cdzl cdfl 19584 5694 4711 1.209
cdlz cdzl cdfl 19584 5618 4638 1.211
cdzs flac 19584 15094 12821 1.177
cdzs cdfl 2448 5124 4371 1.172
cdlz cdzl cdfl 9792 10922 10903 1.002
lzma zlib huff flac 4096 8430 8427 1.000
zstd flac 4096 10359 10487 0.988
total 86602 80764 1.072

Three independent GD-ROM images land within 0.4% of each other, so the CD gain is reproducible rather than one lucky file. The two zstd flac images are the only regression, and both spend 91-95% of their wall time in storage.

On the flash-embedded corpus, where I/O is zero and the measurement is pure CPU: hd_flac 1.233, cd_cdfl 1.198, cd_default 1.179 — and cd_cdlz 0.994, cd_cdzl 0.991, so no regression on the codecs micro-flac does not touch. The slowdown on cdlz-heavy content reported earlier does not survive #188.

RP2350 Cortex-M33, 7 real discs, two runs each with a spread under 0.17%: 1.032x overall, 1.12-1.14x on two images.

Memory. micro-flac's peak is 2-3 KB lower than dr_flac's on 8 of 11 discs, 1-2 KB higher on the rest. Worst-case smallest-largest-free-block - the metric that decided the S3's historical OOMs - is 47 KB for dr_flac against 51 KB for micro-flac.

What it costs

  • include/libchdr/flac.h leaves the installed public headers. The struct differs between backends, so installing it would be actively harmful, but it is a packaging change. No installed header includes it.
  • The synthesised STREAMINFO now exists in two places, src/libchdr_flac.c and src/libchdr_flac_microflac.cpp. They already disagreed once - the .cpp was right before Declare the real FLAC block size in the synthesised STREAMINFO #188 fixed the .c.
  • Apache-2.0 is incompatible with GPLv2, so a GPLv2 consumer cannot enable this backend.
  • The upstream repository is young. The SHA pin, fetch-don't-vendor and CHDR_MICROFLAC_SOURCE_DIR limit the exposure, but it is a real dependency.

Rebased on master (#186, #187, #188, #189); the conflict was the hd_flac corpus seed, which #188 added independently.

CHDR_FLAC_BACKEND=microflac swaps dr_flac for micro-flac behind the same
flac_decoder interface, so exactly one backend is compiled and nothing else in
the tree changes. dr_flac stays the default.

micro-flac is Apache-2.0 and C++, so it is fetched at build time from a pinned
commit rather than vendored: this tree stays BSD-3 for its other consumers, and
a default build neither touches the network nor needs a C++ compiler -
enable_language(CXX) is inside the backend's own branch, and the new workflow
checks that by configuring a default build with a compiler path that does not
exist. Point CHDR_MICROFLAC_SOURCE_DIR at a checkout to build offline.

It fits one thing libchdr needs better than dr_flac does. dr_flac exposes no
way to ask how many input bytes it consumed, so the dr_flac backend
reconstructs that by reaching into the bit-reader's private cache state, and
cdfl uses the answer to find the subcode that follows the audio in the hunk.
micro-flac's streaming decode() returns bytes_consumed directly.

It decodes into the caller's buffer, with no intermediate. micro-flac re-checks
the output buffer against one whole block on every call, sized from the
STREAMINFO block-size field - and the header libchdr synthesises carries
block_size * num_channels where the format specifies inter-channel samples.
Writing the specified value halves what micro-flac demands, which is what makes
the caller's hunk large enough for every block of it. A buffer smaller than one
block, or a final block that would overrun it, still falls back to a retained
scratch; for CD geometry it is never allocated.

That field cuts both ways and the comment says so: micro-flac also *rejects*
any frame whose block size exceeds it, so the smaller value narrows what this
backend accepts. It is safe against what chdman writes, because libchdr derives
the field from the same geometry chdman encoded to, but it is not a margin.

Measured on hardware, at the same point of the same image: 198 KB of heap in
use against dr_flac's 210, and a largest free block of 52 KB against 39 - which
is the figure that decides whether a later allocation of that size succeeds. On
the desktop, peak heap on a cdfl image is 99488 bytes against dr_flac's 121640,
excluding 73728 bytes of libstdc++ static initialisation the dynamic loader
performs before main(); that is a hosted-link artifact and does not exist in a
bare-metal link with the operator-new shim documented in
contrib/tangcore-bl616/README.md.

One incidental correctness gain. dr_flac writes int16_t straight into the
caller's buffer, which is undefined on an odd pointer and is why chd_read
documents a 2-byte alignment requirement. Under UBSan with a deliberately
misaligned buffer, dr_flac aborts on all ten odd-aligned cases and micro-flac
completes every one of them, with no sanitizer diagnostic at all.

Three things this also fixes or closes:

flac.h leaves the installed header set. It is included only from src/, and the
backend makes struct _flac_decoder depend on a macro no consumer defines - so
an installed copy would describe a different layout than the library was built
with.

flac_decoder_free() no longer destroys an object that was never constructed.
chd_close() frees every codec slot, including ones whose init() never ran.

generate.sh gains hd_flac.chd. The raw flac codec was reached only because it
happens to be one of chdman's default hard-disk codecs, and relying on a
default to cover a codec is how a codec stops being covered.

Verified byte-identical against the dr_flac backend throughout: 18 seeds in
three read orders, both settings of CHDR_CD_SCRATCH_BUFFER, and 4253 real
images sampled 400 hunks each. 255 ASan/UBSan runs across seed x access pattern
x alignment. AVHuff's audio streams also use this decoder and are not covered -
that corpus is fetched rather than generated, so wiring it in is its own change.
@rtissera
rtissera force-pushed the microflac-opt-in branch 2 times, most recently from 8616893 to 3079c6c Compare September 6, 2026 13:21
The operator-new shim is a GCC 10.2 artifact - -nostdlib++ replaces it in one
flag and has existed since GCC 11 - and reads as chip-specific when it is not.
Someone will copy it into a project on a modern compiler.

And the reason the vendor toolchain cannot simply be swapped for upstream is
not the T-Head extensions, which GCC has carried since 13. It is -mtune=e907,
the unratified p extension, and zpsfoperand/xtheade, none of which upstream
spells - still true in the GCC 15.2 and 16.1 manuals. Written down so the next
person does not repeat the search.
rtissera and others added 2 commits September 9, 2026 22:41
CMake's regex engine has no {n} repetition, so "^[0-9a-fA-F]{40}$" matches
nothing at all and the guard fired on the correct SHA the workflow passes.

Checked both ways this time: the pinned SHA configures, and "main" is still
refused. The first test only covered the branch name, which failed for the
wrong reason and looked like success.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@rtissera
rtissera merged commit 3349d13 into master Sep 10, 2026
46 of 48 checks passed
@rtissera
rtissera deleted the microflac-opt-in branch September 10, 2026 20:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant