MCU targets: read-ahead budget, micro-flac by default, two dead ends removed

.github/workflows/bl616-tangcore-build.yml

@@ -47,6 +47,10 @@ env:
   TOOLCHAIN_SHA: c4afe91cbd01bf7dce525e0d23b4219c8691e8f0
   BOUFFALO_SDK_SHA: 7f44f9ea6b4ccf96db8c5236c8024b68e2a76df7
   FIRMWARE_BL616_SHA: a5a6ea1cf7c81f32c1c3f0f91ea9d913be5ba078
+  # micro-flac is fetched, never vendored: it is Apache-2.0 and libchdr stays
+  # BSD-3 for its other consumers. Pinned, because the backend compiles
+  # against its internals.
+  MICROFLAC_SHA: ffbe8a9ba5e78c16e535a4695a8b2418b0c091ee
 
 jobs:
   bl616-tangcore-build:
@@ -116,6 +120,11 @@ jobs:
           mkdir -p "$F/chd"
           cp contrib/tangcore-bl616/chd/chd_fatfs.h contrib/tangcore-bl616/chd/chd_fatfs.c "$F/chd/"
 
+          # micro-flac is the default backend for this integration; fetched at a
+          # pinned commit, never vendored into libchdr's own tree.
+          git clone -q https://github.com/esphome-libs/micro-flac.git "$F/thirdparty/micro-flac"
+          git -C "$F/thirdparty/micro-flac" checkout -q "$MICROFLAC_SHA"
+
       - name: Build (TANG_BOARD=console60k)
         run: |
           export PATH="${{ github.workspace }}/tangcore-work/toolchain_gcc_t-head_linux/bin:$PATH"
@@ -124,3 +133,19 @@ jobs:
           make 2>&1 | tee build.log
           echo "---- memory region summary ----"
           grep -A 7 "Memory region" build.log
+
+      # The dr_flac fallback has to keep working for anyone Apache-2.0 does not
+      # suit, and nothing else here would notice if the revert patch rotted.
+      - name: Build again with the dr_flac fallback
+        run: |
+          set -euo pipefail
+          export PATH="${{ github.workspace }}/tangcore-work/toolchain_gcc_t-head_linux/bin:$PATH"
+          F=tangcore-work/firmware-bl616
+
+          git -C "$F" apply "${{ github.workspace }}/contrib/tangcore-bl616/patches/firmware-bl616-libchdr-drflac.patch"
+
+          cd "$F"
+          rm -rf build
+          make 2>&1 | tee build-drflac.log
+          echo "---- memory region summary (dr_flac) ----"
+          grep -A 7 "Memory region" build-drflac.log

contrib/esp32p4/README.md

@@ -36,6 +36,43 @@ directory provides instead:
    `esp_vfs_fat_sdmmc_mount()` for SD/MMC), then call
    `chd_esp_vfs_open("/sdcard/game.chd", &chd)`.
 
+## Options worth setting, and what they measured
+
+`chd_esp_vfs_open()` already calls `chd_set_cache_budget()` with 32KB.
+Compressed hunks are a few KB and laid out sequentially, so one larger read
+serves many and the per-read fixed cost is paid far less often: **1.11x on an
+ESP32-S3** over SPI, **1.05-1.12x on an RP2350** with 32KB the knee there. The
+P4's own number has not been taken. Set `CHD_ESP_VFS_CACHE_BUDGET` to 0 to turn
+it off.
+
+**micro-flac is the default backend here**, unlike libchdr itself. The
+component fetches it at a pinned commit when `CHDR_MICROFLAC_SOURCE_DIR` is not
+given - point that at your own checkout, or at a managed component under
+`managed_components/esphome__micro-flac`, if you would rather not fetch.
+`CHDR_FLAC_BACKEND=drflac` goes back to dr_flac.
+
+Output is byte-identical either way. On an ESP32-S3 with I/O excluded:
+**1.198x** on a CD-FLAC hunk, **1.233x** on raw FLAC. Across eleven real discs:
+**1.072x** overall, 1.21x where the image is FLAC-heavy, **0.988x** on one
+profile where FLAC barely appears. Peak heap is lower than dr_flac's on most
+images.
+
+libchdr's own default stays dr_flac: a desktop consumer vendoring `src/` must
+not have to fetch anything, and micro-flac is C++ and Apache-2.0. An MCU
+integrator is already cloning an SDK and a toolchain, so one more pinned
+checkout costs nothing - hence the different default on this side.
+
+An earlier revision of this file quoted 1.41x for the P4. That predates the
+STREAMINFO block-size fix, which removed an oversized decoded-sample buffer
+from dr_flac and took most of micro-flac's lead with it. **The P4 has not been
+re-measured since**; treat the S3 numbers above as the estimate until it is.
+
+Two things not to try, both with their numbers in
+`../../docs/perf-esp32p4-findings.md`: **`-Os`** is 1.12x *slower* than the
+`PERF` (-O2) default on an S3, and **`Z7_LZMA_PROB32`** costs 15,980 bytes per
+LZMA instance for a speedup the LZMA SDK only claims for "some CPUs" and that
+was never measured on any target.
+
 ## CI
 
 Two workflows:

contrib/esp32p4/chd/chd_esp_vfs.c

@@ -67,7 +67,22 @@ chd_error chd_esp_vfs_open(const char *path, chd_file **chd)
 
 	err = chd_open_core_file_callbacks(&chd_esp_vfs_callbacks, f, CHD_OPEN_READ, NULL, chd);
 	if (err != CHDERR_NONE)
+	{
 		fclose(f);
+		return err;
+	}
 
-	return err;
+	/* Compressed hunks are small - a few KB - and laid out strictly
+	 * sequentially, so one larger read serves many of them and the fixed cost
+	 * of each read (VFS dispatch, FATFS bookkeeping, controller command setup,
+	 * DMA, interrupt) is paid far less often. Measured 1.11x on an ESP32-S3
+	 * reading over SPI, and 1.05-1.12x on an RP2350 with 32KB the knee there.
+	 *
+	 * A ceiling, not an allocation request: an image whose hunks exceed it
+	 * leaves caching off rather than over-allocating, and failing to set it is
+	 * not fatal to the open. */
+	if (CHD_ESP_VFS_CACHE_BUDGET != 0)
+		(void)chd_set_cache_budget(*chd, CHD_ESP_VFS_CACHE_BUDGET);
+
+	return CHDERR_NONE;
 }

contrib/esp32p4/chd/chd_esp_vfs.h

@@ -32,6 +32,13 @@ extern const core_file_callbacks chd_esp_vfs_callbacks;
 /* Opens path via fopen("rb"), then hands it to libchdr as the argp for
  * chd_esp_vfs_callbacks. On any failure, the FILE* is closed if it was
  * opened and *chd is left untouched. */
+/* Read-ahead window handed to chd_set_cache_budget() on every open. Set to 0
+ * to leave it off. 32KB is the knee measured on the boards where this was
+ * tried; the ESP32-P4's own number has not been taken. */
+#ifndef CHD_ESP_VFS_CACHE_BUDGET
+#define CHD_ESP_VFS_CACHE_BUDGET (32 * 1024)
+#endif
+
 chd_error chd_esp_vfs_open(const char *path, chd_file **chd);
 
 #ifdef __cplusplus

contrib/esp32p4/idf-benchmark/components/libchdr/CMakeLists.txt

@@ -20,18 +20,51 @@ endif()
 
 set(LIBCHDR_ROOT "${CMAKE_CURRENT_LIST_DIR}/../../../../..")
 
-# FLAC backend, mirroring libchdr's own CHDR_FLAC_BACKEND. dr_flac is the
-# default. microflac is measurably faster here - 1.46x on the S3 and 1.41x on
-# the P4 decoding CD audio - but it is C++ and Apache-2.0, so it is not
-# vendored: point CHDR_MICROFLAC_SOURCE_DIR at a checkout, or add
-# esphome/micro-flac to the project's idf_component.yml and point this at
+# FLAC backend. micro-flac is the default here, unlike libchdr itself: on an
+# ESP32-S3 with I/O excluded it is 1.198x on a CD-FLAC hunk and 1.233x on raw
+# FLAC, its peak heap is lower on most images, and an MCU integrator already
+# clones an SDK and a toolchain, so one more pinned checkout costs nothing.
+# libchdr's own default stays dr_flac: a desktop consumer vendoring src/ must
+# not have to fetch anything.
+#
+# The ESP32-P4 has not been re-measured since the STREAMINFO block-size fix,
+# which took an oversized buffer out of dr_flac and most of micro-flac's lead
+# with it - the older 1.41x quoted here is not valid.
+#
+# Set CHDR_FLAC_BACKEND=drflac to go back, e.g. if Apache-2.0 does not suit
+# (micro-flac is Apache-2.0; libchdr stays BSD-3 because this is fetched, never
+# vendored). Or point CHDR_MICROFLAC_SOURCE_DIR at your own checkout, including
+# a managed component at
 # ${CMAKE_BINARY_DIR}/../managed_components/esphome__micro-flac.
 if(NOT DEFINED CHDR_FLAC_BACKEND)
+	set(CHDR_FLAC_BACKEND microflac)
+endif()
+
+# ESP-IDF runs every component CMakeLists in script mode first, just to collect
+# REQUIRES, and FetchContent is not scriptable there (define_property fails).
+# The backend choice does not affect requirements, so that pass takes the
+# in-tree source and the fetch happens once, in the real configure.
+if(CMAKE_SCRIPT_MODE_FILE)
 	set(CHDR_FLAC_BACKEND drflac)
 endif()
+set(CHDR_MICROFLAC_GIT_TAG "ffbe8a9ba5e78c16e535a4695a8b2418b0c091ee"
+	CACHE STRING "micro-flac commit to fetch when no local checkout is given")
 if(CHDR_FLAC_BACKEND STREQUAL "microflac")
 	if(NOT CHDR_MICROFLAC_SOURCE_DIR)
-		message(FATAL_ERROR "CHDR_FLAC_BACKEND=microflac needs CHDR_MICROFLAC_SOURCE_DIR")
+		# Pinned SHA, because the backend compiles against micro-flac's
+		# internals. Fetched at configure time rather than through
+		# idf_component.yml so that this component stands alone.
+		include(FetchContent)
+		# SOURCE_SUBDIR names a directory that does not exist on purpose:
+		# it downloads the sources without running micro-flac's own
+		# CMakeLists, whose targets would otherwise be configured for the
+		# host and joined to this cross build.
+		FetchContent_Declare(microflac
+			GIT_REPOSITORY https://github.com/esphome-libs/micro-flac.git
+			GIT_TAG ${CHDR_MICROFLAC_GIT_TAG}
+			SOURCE_SUBDIR chdr-sources-only)
+		FetchContent_MakeAvailable(microflac)
+		set(CHDR_MICROFLAC_SOURCE_DIR "${microflac_SOURCE_DIR}")
 	endif()
 	# micro-flac enables its Xtensa LPC assembly whenever the core has the
 	# required features, so the .S files must be compiled in on esp32/esp32s3

contrib/rp2350/CMakeLists.txt

@@ -35,22 +35,40 @@ if(NOT DEFINED SD_BAUD_HZ)
 	set(SD_BAUD_HZ 25000000)
 endif()
 
-# The LZMA SDK ships decoder assembly for x86-64 and AArch64 only; Cortex-M33
-# is ARMv8-M Thumb, so neither applies. Z7_LZMA_PROB32 is the one portable
-# knob: it widens the probability model from UInt16 to UInt32, which the SDK
-# notes "can increase the speed on some CPUs" at double the probs memory
-# (+15,980 bytes per LZMA instance at CHD's lc=3/lp=0). Off by default.
-option(CHDR_LZMA_PROB32 "Use 32-bit LZMA probability model" OFF)
 
-# FLAC backend, mirroring libchdr's own CHDR_FLAC_BACKEND option. dr_flac is
-# the default; microflac needs CHDR_MICROFLAC_SOURCE_DIR pointing at a checkout.
+# FLAC backend. micro-flac is the default here, unlike libchdr itself: it is
+# measurably faster on every board this was tried on (1.198x on a CD-FLAC hunk
+# with I/O excluded on an ESP32-S3, 1.12-1.14x on two real discs here), its
+# peak heap is lower on most images, and its worst-case largest-free-block is
+# better - and an MCU integrator is already cloning an SDK and a toolchain, so
+# one more pinned checkout costs them nothing. libchdr's own default stays
+# dr_flac, because a desktop consumer vendoring src/ must not have to fetch
+# anything.
+#
+# Set CHDR_FLAC_BACKEND=drflac to go back, e.g. if Apache-2.0 does not suit
+# (micro-flac is Apache-2.0; libchdr stays BSD-3 because this is fetched, never
+# vendored).
 if(NOT DEFINED CHDR_FLAC_BACKEND)
-	set(CHDR_FLAC_BACKEND drflac)
+	set(CHDR_FLAC_BACKEND microflac)
 endif()
+set(CHDR_MICROFLAC_GIT_TAG "ffbe8a9ba5e78c16e535a4695a8b2418b0c091ee"
+	CACHE STRING "micro-flac commit to fetch when no local checkout is given")
 if(CHDR_FLAC_BACKEND STREQUAL "microflac")
 	enable_language(CXX)
 	if(NOT CHDR_MICROFLAC_SOURCE_DIR)
-		message(FATAL_ERROR "CHDR_FLAC_BACKEND=microflac needs CHDR_MICROFLAC_SOURCE_DIR")
+		# Same shape as libchdr's own root CMakeLists: pinned SHA, because the
+		# backend compiles against micro-flac's internals.
+		include(FetchContent)
+		# SOURCE_SUBDIR names a directory that does not exist on purpose:
+		# it downloads the sources without running micro-flac's own
+		# CMakeLists, whose targets would otherwise be configured for the
+		# host and joined to this cross build.
+		FetchContent_Declare(microflac
+			GIT_REPOSITORY https://github.com/esphome-libs/micro-flac.git
+			GIT_TAG ${CHDR_MICROFLAC_GIT_TAG}
+			SOURCE_SUBDIR chdr-sources-only)
+		FetchContent_MakeAvailable(microflac)
+		set(CHDR_MICROFLAC_SOURCE_DIR "${microflac_SOURCE_DIR}")
 	endif()
 	set(CHDR_FLAC_SRC
 		${LIBCHDR_ROOT}/src/libchdr_flac_microflac.cpp
@@ -120,11 +138,6 @@ target_compile_definitions(rp2350-bench PRIVATE
 	MINIZ_NO_TIME
 )
 
-if(CHDR_LZMA_PROB32)
-	set_source_files_properties(${LIBCHDR_ROOT}/deps/lzma-26.02/src/LzmaDec.c
-		PROPERTIES COMPILE_DEFINITIONS Z7_LZMA_PROB32)
-endif()
-
 target_compile_options(rp2350-bench PRIVATE -O3)
 target_link_libraries(rp2350-bench
 	pico_stdlib

contrib/rp2350/README.md

@@ -78,6 +78,22 @@ corpus. No RP2350 hardware in CI, so this can't prove `chd_open()`/
 neither build regresses correctness or blows past budget on what CI *can*
 see.
 
+## FLAC backend: micro-flac by default
+
+The benchmark here decodes FLAC through
+[micro-flac](https://github.com/esphome-libs/micro-flac) rather than dr_flac,
+fetched at a pinned commit unless `CHDR_MICROFLAC_SOURCE_DIR` points at a local
+one. `-DCHDR_FLAC_BACKEND=drflac` goes back.
+
+Output is byte-identical. Measured on this board across seven real discs,
+two runs each with a spread under 0.17%: **1.032x** overall, **1.12-1.14x** on
+two of them, and a worst-case largest-free-block of 51 KB against dr_flac's
+47 KB. On an ESP32-S3 with I/O excluded it is 1.198x on a CD-FLAC hunk.
+
+libchdr's own default stays dr_flac - a desktop consumer vendoring `src/`
+should not have to fetch an Apache-2.0 C++ dependency. An MCU integrator is
+already cloning an SDK, so the trade is different here.
+
 ## Real-hardware benchmark (this directory)
 
 The CI workflows above run under QEMU and measure RAM only; as their note