Skip to content

packing/uepak: read version 9 frozen pak indices - #2476

Open
YoungTreeTree wants to merge 2 commits into
seedds:mainfrom
YoungTreeTree:fix/uepak-v9-frozen-index
Open

YoungTreeTree wants to merge 2 commits into
seedds:mainfrom
YoungTreeTree:fix/uepak-v9-frozen-index

Conversation

@YoungTreeTree

@YoungTreeTree YoungTreeTree commented Nov 12, 2026 •

Copy link
Copy Markdown

What and why

Pak v9 archives were skipped because their footer adds a FrozenIndex byte and moves the magic to 0xcd bytes from EOF. When that flag is set, accepting the version alone is insufficient: the index is a frozen UE4 memory image.

This reads bounded relative pointers, UTF-16 strings, FPakEntry arrays and nested sparse directory maps, retaining entry-relative compression blocks and the existing serialized-index path when FrozenIndex is clear. Encrypted-index key discovery checks the pointed-to mount. Shader discovery reads the directory before distinguishing inline/global-cache paks, shared shader archives and audio-only paks.

This addresses the Pak parsing/extraction failure reported in #2134. Related: #1618 handles ordinary v9 indices but explicitly rejects frozen indices; the Demon Slayer archives tested here use FrozenIndex=1.

Finding: Unreal Engine frozen pak indices (version 9).

Checked

  • On the updated branch (ce370e1 base): all 63 test_uepak*.py tests (including eight new v9 tests), all 6 documentation tests, and the pinned Python lint ratchet pass (zero new findings).
  • Synthetic coverage includes both index layouts, sparse-map holes, signed backward pointers, UTF-16 names, multi-block Zlib data, invalid ranges/references, deleted entries, encrypted-index key search and shader classification. No game-derived fixtures.
  • Real archives: 1,284 + 88 entries agree with their serialized data headers; sampled configuration/audio and the 8,191,424-byte global shader cache decode to the indexed sizes.
  • The shader split finds 57,982 shaders, with 1,519 first found in the pak's global cache. All 57,982 build for the Mac; the host and pipeline prebuild complete.
  • A 45-second bounded random parser-fuzz run, adding the synthetic v9 pak to the existing harness seeds, completed 414,168 executions with no findings.
  • GPU tests: make -C dm/gfx/tests -j2 GAME=0100309016E7A000 run passed.
  • Game run: Demon Slayer -Kimetsu no Yaiba- The Hinokami Chronicles, 0100309016E7A000, 1.53 / v589824. Confirmed the Simplified Chinese autosave/new-save prompts and reached the title menu with the character-pack notification. Combat, full levels, long-term stability and iOS were not tested. The game remains on a provisional build profile; this does not add it to the supported-games table.
  • Device: Apple M2 Pro, 32 GB, macOS 27.2.1, Xcode 27.2.

The real-game build/run, GPU tests and regression report below were collected on 547c5db plus this parser change before rebasing onto 034e91a. Upstream did not change the pak reader in between; parser/documentation/lint checks were rerun after rebasing. Runtime was not rerun against the newer host.

Regression report

scripts/regress.py check --games 0100309016E7A000: PASS, full mode, 324.7 seconds.

Baseline 547c5db647a2fc3880661d1357041d4dd1aae371; candidate the same revision plus this patch.

game status shaders (base → candidate) newly failing newly building changed MSL interface (new / known) automated boot
Demon Slayer 0100309016E7A000 PASS 57982/57982 → 57982/57982 0 0 0 0 / 0 —

This was the only local game build available and directly exercises frozen v9 paks. Other games are not runtime-validated; existing v4/v8/v11 synthetic tests cover their parser paths. There is no registered smoke route for this provisional profile, so the report's boot/performance fields are empty; the manual menu observation above is separate. The harness compares the prepared shader package, while the real-archive and unit checks validate extraction.

CI checks

The branch is based on ce370e1. The save-sync runner-model check now comes from upstream; this PR no longer changes that test.

A separate test-only commit addresses the clang-tidy failures in the graphics tests: check nullable Metal resources before use; initialize and check the Vulkan swap chain; check the wide-line fixture paths and run its translator through NSTask; and free the sparse-copy test's linear command records after submission. The full LLVM run also regenerates the shrinking baseline to remove resolved findings. Runtime and shader-chain code are unchanged by this CI commit.

  • All seven affected GPU tests pass on the M2 Pro with Xcode 27.2: t_res_commit (310 checks), t_upload_src (41), t_zero_depth (56), t_sparse_copy (43), t_vk_swapchain_life (141), t_wline (22), and t_pfv (zero failed). The latter four also ran with Metal API validation enabled.
  • All 63 pak tests, 6 documentation tests and 19 lint-tool tests pass. The pinned Python lint ratchet reports 321 existing findings, zero new findings.
  • LLVM 23.1.4 full clang-tidy validation passes across 312 sources: 204 findings, exactly matching the baseline, zero new findings and zero stale counts. The baseline shrank from 211 to 204 via lint_c.py tidy --update; no checks were disabled or findings added. Final check: python3 dm/tools/lint_c.py tidy -j 6 --against ce370e1.

The real-game and shader regression evidence above predates this rebase. No new game-runtime or iOS validation is claimed for the updated base.

Docs

  • Reference finding added and python3 dm/tools/docs_symptoms.py run.
  • No game files, keys or extracted game data in the change or attachments.
  • Shader translation-chain inputs are unchanged; no chain-version bump is needed.

@github-actions github-actions Bot added tools Tools and scripts (dm/tools, scripts) docs Documentation host The Horizon host: SVCs, services, threads, memory (dm/host) labels Nov 12, 2026
seedds and others added 2 commits November 14, 2026 23:22
Pak v9 moves the footer magic by one byte and can store its index as a
frozen UE4 memory image. The reader previously skipped these archives,
leaving Demon Slayer's shader split empty.

Read the FrozenIndex flag and bounded relative pointers, UTF-16 strings,
entry arrays and sparse directory maps. Preserve the serialized v9 index
path and recognize encrypted frozen mounts during key discovery. Classify
audio-only paks separately from shader archives after reading the index.

Synthetic tests cover both layouts, sparse holes, backward pointers,
multi-block compression, encrypted indices and malformed references.
Both real game indices validate, all 57,982 shaders build, and an M2 Pro
run reaches the Simplified Chinese title menu. Combat is not tested.

Related to seedds#2134 and seedds#1618.

Co-authored-by: YoungTreeTree <24680817+YoungTreeTree@users.noreply.github.com>
The CI clang-tidy run reports unchecked nullable Metal resources in graphics
tests. Check their buffers, views and shader functions before use. Initialize
the swap chain with valid enum values and check its creation result before
reading the returned images. Check the wide-line fixture's file paths and
invoke its Python translator with arguments through NSTask.

Retain the sparse-copy test's linear command records until submission, then
free them, matching the production queue's ownership. Regenerate the shrinking
lint baseline with LLVM 23.1.4 so it no longer lists resolved findings.

All seven affected GPU tests pass on an M2 Pro with Xcode 27.2. The four
additional tests also pass with Metal API validation enabled. The 63 pak
tests, documentation tests and pinned Python lint ratchet pass as well.
@YoungTreeTree
YoungTreeTree force-pushed the fix/uepak-v9-frozen-index branch from 0646558 to f694d4d Compare November 14, 2026 15:48
@github-actions github-actions Bot added the nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) label Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation host The Horizon host: SVCs, services, threads, memory (dm/host) nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) tools Tools and scripts (dm/tools, scripts)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants