diff --git a/docs/dev/airdefense1-optimization-evidence.md b/docs/dev/airdefense1-optimization-evidence.md index 6a787239b..068fa6c99 100644 --- a/docs/dev/airdefense1-optimization-evidence.md +++ b/docs/dev/airdefense1-optimization-evidence.md @@ -108,3 +108,175 @@ asset compatibility rather than an overlay-free executable. Representative OpenGL/Vulkan SP, screen-effect, and MP engine screenshots were reviewed for black output, invalid geometry, missing HUD, and obvious presentation defects; none were found. + +## 2026-08-31 camera-sweep pass + +This pass added a repeatable camera-motion capture for the same scene, then ran +optimization and robustness rounds against it. All runs are Windows x64, +bordered windowed, OpenGL, uncapped presentation, shipping renderer defaults, +and stock retail assets. No mouse or keyboard input was synthesized. + +### The sweep capture + +Measuring a fixed spawn view only exercises one frustum. `benchmarkViewSweep` +pans the local player's view a requested arc over a requested duration, driven +entirely from game time inside `idPlayer::UpdateViewAngles`, so the same sweep +is reproduced regardless of host frame rate and no operating-system input is +involved. The completion line reports the arc it actually walked. + +The acceptance capture skips the loading-screen continue gate +(`com_skipLoadingContinue 1`) and the opening cinematic +(`g_autoSkipCinematics 1`), settles, then samples a full turn: + +``` +python tools/tests/renderer_gameplay_benchmark.py --cases sp-airdefense1 \ + --tiers auto --maxfps 0 --swap-intervals 0 --display-modes windowed \ + --render-api gl --pacing-only --no-gpu-timers --settle-frames 360 \ + --exec-command "benchmarkViewSweep 360 12000" --sample-msec 13000 +``` + +### Frame rate over a full 360-degree turn + +Three consecutive passing captures on the final build at 1280x720, each +sampling one complete turn from the starting area: + +| Run | Samples | Average | P50 | P95 | P99 | Worst frame | +|---|---:|---:|---:|---:|---:|---:| +| 1 | 3,472 | 284.1 Hz | 4 ms | 6 ms | 7 ms | 12 ms | +| 2 | 3,544 | 298.7 Hz | 4 ms | 6 ms | 7 ms | 11 ms | +| 3 | 3,363 | 282.8 Hz | 4 ms | 6 ms | 7 ms | 8 ms | + +Median 284.1 Hz, and six passing captures taken across this pass span +282.8--313.1 Hz with an unchanged 6 / 7 ms P95 / P99. The worst single +frame in any capture was 14 ms, so the turn holds well above the 100 FPS +acceptance target throughout rather than averaging over a stall. The same +sweep at 1920x1080 reported 310.8 Hz with an +identical 6 / 7 ms P95 / P99 and a 10 ms worst frame, which matches the earlier +finding that this scene is CPU/front-end limited rather than fill limited. +Every run reported the sweep completing its full arc, and the end-of-sweep +engine screenshots show the expected geometry, lighting, and HUD. + +### Where the load time goes + +Load timings need a save path that survives between runs. The gameplay +benchmark deliberately requires a fresh output directory, so each of its runs +regenerates the binary image cache and reports a first-visit load. A player +pays that once. Warm medians of three runs, stock defaults: + +| Phase | msec | +|---|---:| +| Game media precache (models, sounds, animations, entity defs) | 4,140 | +| Level image loading (1,718 files, 213 MiB) | 2,538 | +| Cinematic fast-forward, only paid when the opening cinematic is skipped | 1,876 | +| Render-world `.proc` parse | 1,355 | +| Collision `.cm` parse | 805 | +| Player spawn and remainder | ~570 | +| **Total** | **11,282** | + +The load is almost entirely single-threaded CPU work: one complete run measured +18.8 s of process CPU against 21.8 s of wall time on a 20-core host, so roughly +nineteen cores are idle throughout. Parallel asset decode, not a faster parser, +is the structural lever here. + +### Optimizations landed + +Both fixes are in the level-load cache manager and are behaviour neutral. + +1. **Semantic-hint matching was quadratic.** Every opened source was matched + against the whole recorded hint vector, and each comparison built up to + three temporary `std::string`s. `game/airdefense1` records 4,428 hints, so + the recording pass scaled as sources times hints. Hints are now indexed by + the three keys the match rules actually use (exact name, compiled-suffix + name, extension-stripped stem), and lookup takes the highest matching hint + index, which is the hint the old reverse scan stopped on. +2. **Each semantic record re-opened its source.** A name already resolved in + the current generation is no longer resolved a second time; the repeat open + could only yield the identity the first one already learned. + +Measured on the level-load cache path (`com_levelLoadModernization 1`), warm +medians of three runs: + +| | Before | After | Change | +|---|---:|---:|---:| +| Total load | 11,893 ms | 9,010 ms | -2,883 ms (-24.2%) | +| Level image loading | 4,442 ms | 2,209 ms | -2,233 ms (-50.3%) | +| First-visit total | 21,040 ms | 18,044 ms | -2,996 ms (-14.2%) | + +### `com_levelLoadModernization` remains default off + +With the fixes above, that path is now the fastest warm configuration measured +for this map: 9,010 ms against 11,282 ms for the shipping default, a 20.1% +reduction, driven by the generated world (1,355 -> 392 ms) and collision +(805 -> 152 ms) caches. + +It is still slower on a first visit: 18,044 ms against 11,418 ms, because that +visit both writes the generated caches and learns the replay manifest. Setting +`com_levelLoadCacheWrite 0` only recovers about 1.3 s of that, so the remaining +first-visit cost is the cache-miss and identity resolution work rather than the +writes themselves. + +That trade -- roughly 6.6 s worse once against 2.3 s better every time after -- +is exactly the cold/warm qualification this feature's promotion gate is waiting +on, so the default is unchanged. Reducing the first-visit cost is the work that +would justify promoting it. + +### Robustness + +- The classic `.proc` `ParseModel` path now range-checks its file-provided + vertex and index counts and every index it reads. `ParseShadowModel` and the + binary render-world cache already applied those predicates; the text draw + surface path did not, and `FinishSurfaces` dereferences every index while + deriving tangents and silhouette edges. Verified against `game/airdefense1`, + `game/airdefense2`, `game/storage1`, `game/medlabs`, and `game/mcc_landing`: + no stock surface is rejected. +- `idImageManager::LoadLevelImages` bounds its fill loop against the array it + pre-sized from `CountPendingLevelLoads`. The two share a predicate today, so + the bound cannot trip; without it a future divergence would be a silent heap + overflow rather than a dropped image. +- `benchmarkViewSweep` clamps both console arguments, so a typo cannot produce + a NaN yaw or a sweep that never ends. + +### Diagnostics + +- `g_frametime` now prints the per-frame section breakdown that previously only + the airdefense1 skip probe could collect (view setup, AI, PVS, network event + queue, gravity, BSE start/end, active-list sort). This is what identified the + 1.85 s "first settle frame" as the cinematic fast-forward loop -- roughly + 11,600 simulation ticks inside one host frame -- rather than a slow frame. +- The sweep's completion line reports the yaw it started from and reached, so a + capture proves the camera panned instead of trusting the request. + +### Font parity failure found while validating, and fixed + +`renderer_validation_matrix` failed its `renderer-foundation-selftests` case +on `uiFontParitySelfTest`: the HUD radio marine cases were a pixel out in line +height, baseline, glyph y, and overhang, the three radio strings were two +pixels narrow, and the loading title measured 234 px against a retail 223 px, +moving its right-aligned x by 11 px. + +It was not from the optimization work -- rebuilding with those engine changes +reverted reproduced it exactly -- but from `SetFontByScale` choosing its atlas +from a viewport-enlarged scale. That selection is also what `TextWidth`, +`MaxCharHeight`, and the `DrawText` advances read, so above roughly 1.5x +enlargement the hand-authored retail `.fontdat` atlases started reporting +different metrics. It reproduced only on a large window, which is why 1280x720 +and 640x480 both passed and the user's own 2538x1312 window did not. + +A scalable font rasterises its three slots from one face at 12/24/48 point, so +their normalised metrics agree and a larger slot only adds resolution. The +enlargement now applies to those fonts only; the retail atlases select on the +authored scale and keep exact parity. The matrix is 36/36 again. + +### Ranked remaining work + +1. Parallel level image decode. 1,718 files and 213 MiB of mostly independent + inflate and decode work currently run serially ahead of the serial GL + upload. This is the largest single remaining item and the one the idle core + count most clearly supports. +2. First-visit cost of the level-load cache path. Closing that gap is what + turns `com_levelLoadModernization` from a warm-only win into a promotable + default. +3. Preload replay coverage. The pipeline admits 64 of 4,428 learned sources and + peaks at 41 MiB of its 384 MiB staging budget, because each admitted source + holds an open file handle for the whole load. Lazily opening handles as the + pipeline consumes them would let the cap rise. diff --git a/docs/dev/releases/v0.12.0.md b/docs/dev/releases/v0.12.0.md index 374e02ce2..5bfcd6c55 100644 --- a/docs/dev/releases/v0.12.0.md +++ b/docs/dev/releases/v0.12.0.md @@ -4,14 +4,15 @@ - **High-refresh gameplay looks smoother without changing game logic.** The first-person camera and weapon, skeletal animation, projectiles, eligible moving-world entities, lights, client effects, and moving-world attachments now sample a presentation pose between authoritative game ticks. Animated cockpits, held weapons, and joint-bound effects stay aligned with the pose actually drawn, including the lightning gun beam, while simulation, networking, collision, demos, and savegames retain their original timing. Multiplayer actor bodies remain on the network/simulation clock; this includes the local body and world-weapon stencil shadows visible in first person. - **Experimental temporal AA can keep the 3D scene steadier while resolution follows GPU load.** `r_temporalAA 1` enables native-history TAA/TAAU on OpenGL or Vulkan, and `r_rendererDynamicResolution 1` adds a delayed, non-blocking GPU-time controller. HUD and menus stay native-sized, camera cuts and captures start from clean history, unsupported moving effects are rejected conservatively, and the established SMAA path remains one setting away. -- **Repeat level-load caching is available as a guarded experiment without slowing ordinary play.** Classic source loading is again the default after testing found that cache preparation could make some stock loads much longer. Developers can opt in with `com_levelLoadModernization 1`; every reuse remains tied to exact source and runtime identity, while stale or corrupt data falls back to the installed source automatically. +- **Repeat level-load caching is available as a guarded experiment without slowing ordinary play.** Classic source loading is again the default after testing found that cache preparation could make some stock loads much longer. Developers can opt in with `com_levelLoadModernization 1`; every reuse remains tied to exact source and runtime identity, while stale or corrupt data falls back to the installed source automatically. Repeat visits through that path are now considerably quicker than in earlier builds: on the complex `airdefense1` opening a return visit loads about a fifth faster than the classic default. The first visit to a map is still slower while the cache is prepared, which is why the setting stays off by default. - **Experimental GPU animation is available without changing gameplay geometry.** Capable OpenGL and Vulkan systems can opt into an exact four-weight MD5/MD5R deformation path with `r_gpuSkinning 1`. Collision, hits, decals, overlays, and stencil shadows retain their CPU-owned data, authored vertex colors remain intact, and any unsupported model or resource condition falls back to the complete CPU path. - **Material authors can evaluate guarded PBR lighting and bounded authored reflections without changing retail materials.** Dual-authored `pbr { ... }` materials retain their normal Quake 4 fallback while eligible metallic/roughness data can use the experimental OpenGL G-buffer, deferred, and clustered-forward route when `r_rendererModernVisible 1` is also enabled. OpenGL authors can opt into specular probes bounded to eight cubemaps, 32 records, and the best two probes per cluster; incomplete probe data returns to the analytic environment. A separate default-off clustered-decal corridor transfers only complete sealed subsets. Vulkan remains limited to a narrow opaque packed-ORM direct-light route whose classic fallback has exactly one active bump -> diffuse -> specular interaction sequence. The local Windows current-source implementation gate passed across stock SP/MP, forced GL 3.3--4.5, Vulkan, and exact master rollback. These Milestone F paths remain experimental and default-off; final committed-package, platform/driver, and release promotion remain pending. - **Players can independently preview bounded volumetric atmosphere, reflections, and indirect light on either renderer.** Default-off froxel volumetrics, SSR, and SSGI now share the native scene presentation path on OpenGL and Vulkan. Each has a fixed work ceiling and its own quality controls; `r_rendererModernQuality 0` rolls all experimental Milestone F lighting back at once. These are deliberately screen-space approximations, so off-screen reflections, per-light volumetric shadows, and world-space multi-bounce GI are outside this release. - **The complex `airdefense1` opening reaches gameplay faster and its CPU-limited scene runs more smoothly.** Cinematic fast-forward no longer spends time interpolating thousands of poses that cannot be displayed, and ordinary maps without baked light grids no longer build a bake-only probe layout while loading. The guarded learned-cache experiment remains off by default; these gains apply to the classic source path. - **Shadow-mapped props now cast complete, grounded shadows.** Open and thin stock models such as the buggy, crates, railings, and computer props around the `airdefense1` start no longer disappear from point-light shadow maps, while sealed meshes store their near surface instead of a detached far shell. Tighter balanced filtering and a world-space cap for huge point-light bias preserve contact corners; any caster that cannot enter the map is supplied by the matching stencil ownership or safely returns that ownership to full stencil. +- **Damaged or incomplete map data is refused instead of trusted.** Compiled world geometry with impossible surface counts or out-of-range triangle data now stops with a clear message naming the problem, matching how the engine already treats compiled shadow geometry. Every stock Quake 4 map continues to load unchanged. - **Stock maps no longer report or expose known startup gaps.** The engine first honours any loose or mod-provided versions of the three omitted brown-fluid effect images and the omitted large water-splash sample, then uses shape- and family-compatible media that did ship with Quake 4. Generated TrueType atlases also bind their uploaded images on first parse, wide loading backgrounds and generated image caches remain reliable from deeply nested save locations, and rigid bodies that the stock game deliberately clamps or forces to rest remain visible as developer diagnostics without being mislabeled as unresolved warnings. -- **Released v0.10 saves no longer depend on whether the game module was optimized.** The exact affected Windows x64 save layout now restores its omitted physics frame and two later player liquid-state fields safely, while new saves use one deterministic class-frame layout in debug and release builds. +- **Saves from older 0.12 development builds load again, and saves that cannot load are refused before they cost you the level you were playing.** Two player liquid fields were added to the save six days apart, but a single check decided both were present, so saves written in between read a field their file does not contain and failed part way through restoring — after the running map had already been unloaded. Each field is now matched to the build that introduced it. Saves too old for this build are turned away immediately, with a message naming which build wrote them, and you stay in the game you were playing. New saves use one deterministic class-frame layout in debug and release builds. - **Polish and Russian are now built in.** Text rendering works by Unicode code point, the Polish tables use the correct Central European encoding path, and generated font data is checksum-validated so translated menus and HUD text remain reproducible across packages. - **The marine hovertank has its vehicle audio back.** Single-player once again creates and updates the engine and hover-pad loops used throughout the vehicle sequence, with safe cleanup during map or engine shutdown. - **Long macOS sessions keep their sound instead of dropping back to the menu.** Apple Silicon packages now include OpenAL Soft, avoiding the fixed source and buffer limits in Apple's legacy OpenAL framework that could stop stock levels after sustained audio use. @@ -20,19 +21,21 @@ - **Multiplayer function keys work as shown.** F1/F2 voting, F3 ready-up, F6 team switching, and F7 spectating once again use Quake 4's real impulse actions. Ready presses travel over the reliable path, the two-line warmup instruction remains fully visible, and only the exact older openQ4 defaults are upgraded—custom bindings stay untouched. No-time-limit deathmatch also starts without a spurious competitive-rules rejection. - **Pausing single-player is immediate.** Opening the in-game menu no longer blocks on save, mod, device, display, key-binding, or multiplayer-model discovery. Those lists refresh only when their own page is opened, and the normal level-loading phase prepares the menu art, fallback image, and music before the first Escape press. - **Single-player console tuning remains under the player's control.** The weapon wheel now uses a dedicated transient slow-motion channel instead of rewriting `timescale`, so values such as `timescale 0.5` remain set after the console closes. The old forced-run shortcut is also gone: `pm_walkspeed` once again controls actual walking when Always Run is off (or the run key temporarily inverts it). The wheel still slows simulation and audio while held, then restores only its own temporary effect. -- **Menu and HUD text stays sharp on modern displays.** Bitmap fonts now choose their source atlas using the final viewport enlargement, preventing a small 640x480-era atlas from being magnified at 1080p, 1440p, or 4K. Existing text layout is unchanged, and manually customized font-limit CVars still take precedence. +- **Smoke and effects no longer cut a hard line where they meet the world.** Soft particles are on by default now. Eligible effect sprites fade out as they approach solid surfaces instead of ending in a visible straight edge against a floor, wall, or crate, which is most obvious in the smoke and steam of scenes like the `airdefense1` opening. Decals, beams, electricity, trails, model debris, and custom-shader effects are untouched, and the fade width stays adjustable with `r_softParticleFadeDistance`. +- **Menu and HUD text stays sharp on modern displays.** Scalable fonts now choose their source atlas using the final viewport enlargement, preventing a small 640x480-era sheet from being magnified at 1080p, 1440p, or 4K. Text layout is unchanged: all three sizes of a scalable font come from one typeface, so a larger one only adds resolution. The original Quake 4 bitmap atlases were drawn by hand at each size and do not agree that exactly, so they keep their authored size and their original spacing, line height, and alignment. Manually customized font-limit CVars still take precedence. - **Developer map compilation is more reliable.** `dmap` now resolves editable Quake 4 `func_group` entities into world geometry before compilation, matching the retail tool and fixing false leaks such as the `hangar1` report in discussion #124. Engine-side geometry tools also initialize their triangle-surface allocator before generating light volumes, fixing the Windows crash seen while compiling `game/airdefense1`. - **Liquids now look, sound, and behave like real volumes in combat.** Projectiles and hitscan shots cross the surface with a splash and sound, underwater travel produces bubble trails, clear water stays readable, and swimmers can reliably water-jump out over a clear ledge. Drowning, slime, and lava have distinct localized death-feed icons and messages. The multiplayer Liquid Volume Lab makes everything easy to inspect across deep, shallow, and wading water, vivid cellular slime, and bright heat-hazed, steaming lava, with restrained underwater ambience and boiling hazardous surfaces. - **Internet multiplayer has safer administration, package negotiation, packet handling, and pure play.** The default remote console no longer sends its password over the network, private password settings stay out of console completion and persistence paths, and unauthenticated replies are bounded. A server can now update only the userinfo or synchronized-setting class owned by each message, and a truncated settings dictionary changes nothing. Server-provided package links are limited to bounded HTTP or HTTPS URLs with syntax-checked hosts; validated web redirects remain available, while standard packages keep in-process direct PK4 transfer disabled. Truncated covered messages and snapshots now fail safely and a malformed snapshot ends the affected session before another frame, the obsolete executable updater can no longer download or launch code, and pure servers keep asset checks enabled while game modules remain trusted local package components rather than server-supplied downloads. ## Upgrade Notes +- Soft particles (`r_softParticles`) now default to on. An existing configuration keeps whatever value it already recorded, so a config written by an earlier build stays off until you set `r_softParticles 1` in the console or remove that line and let the new default apply. - Replace the complete openQ4 package for your platform so the engine, renderer modules, `game_sp`, `game_mp`, and bundled openQ4 data all come from 0.12.0. Do not mix modules from older releases. - Mac users should replace the complete package to receive the bundled OpenAL Soft runtime; no separate OpenAL installation is needed. The package includes its corresponding licence notices and source offer. - Linux packages no longer contain duplicate root icons whose names differ only by letter case, so the same archive can be extracted safely on Windows and other case-insensitive filesystems. - Existing settings and compatible saves do not need to be reset. As a precaution, retain a copy of important saves before replacing an older installation. - Map authors whose previously sealed Quake 4 maps reported an immediate leak in openQ4 should rerun `dmap` after upgrading; grouped world brushes are now compiled correctly. Keep the original editable `.map` source rather than replacing it with generated output. -- Players upgrading from released v0.10 Windows x64 builds can retry saves that previously stopped at a physics or `ReadTrace` restore error. A modded save still requires the same mod maps, GUIs, and other assets that created it; keep a backup before testing. +- Saves written by openQ4 0.10, and by 0.11-era development builds that used the older version-2 payload, are no longer loadable and are now refused up front instead of failing part way through. Every one of them was tested against this build and none restored correctly, so the support claim was withdrawn rather than left to fail in play. Finish those campaigns on the build that wrote them, or start the chapter again. A modded save still requires the same mod maps, GUIs, and other assets that created it; keep a backup before testing. - The exact older broad shadow-filter defaults are upgraded once to the new balanced contact-shadow values; customized shadow profiles are left untouched. `r_shadowMapCasterCulling 2` remains the recommended value but now means topology-aware automatic culling rather than far-shell storage. Use `0` for an always-two-sided comparison or `1` to force near-shell culling. - Level-load data under the active `fs_savepath` `generated/` directory is private and disposable. `com_levelLoadModernization` defaults to `0` and overrides older archived cache settings, so ordinary play uses the classic source path. Set it to `1` only for focused cache evaluation. Never delete the original retail or mod assets. - Retail Quake 4 assets are still required. Point openQ4 at a complete retail installation, including the unsuffixed base dialogue archive such as `zpak_english.pk4`. Numbered files such as `zpak_english_01.pk4` are patches and must not be renamed as a substitute; without the base archive, dialogue is silent and conversation-gated campaign scenes can fail to advance. @@ -93,7 +96,7 @@ - Resolved Quake 4 `func_group` entities before `dmap` geometry processing, restoring retail brush/entity totals and preventing grouped world geometry from becoming a false leak. - Initialized engine-side triangle-surface allocators for `dmap` and related geometry tools. - Restored clean, fixed-size manual save-game previews at widescreen resolutions and prevented renderer row padding from leaking into narrow captures. -- Restored the exact affected v0.10 Windows x64 version-3 save layout by defaulting its absent swim-speed and liquid-surface timer fields and accepting the empty physics frame omitted by optimized builds; current writers now use source-declared class ownership so linker folding cannot change save bytes. +- Matched each player liquid save field to the build that added it (`swimSpeed` from build 661, `nextLiquidSurfaceSoundTime` from build 721) instead of gating both on one snapshot identity, which recovers version-3 saves written between those builds. Version-3 payloads below the oldest verified build, and all version-2 payloads, are now refused during preflight while the running map is still intact. Current writers use source-declared class ownership so linker folding cannot change save bytes. - Replaced plaintext-by-default remote console authentication with bounded `rcon2`, redacted private password CVars throughout console and persistence paths, restricted server-originated CVar dictionaries to their declared userinfo/network-sync authority with all-or-nothing decoding, constrained server package URLs to bounded HTTP/HTTPS with syntax-checked hosts, and retired executable download/launch behavior from the legacy updater. Standard packages keep direct PK4 transfer disabled; separately integrated curl-enabled builds add time-limited, no-redirect transfer containment. - Restored `si_pure` multiplayer enforcement with ordered asset-PK4 checks and a platform-independent Quake 4 1.4.2 protocol 2.41 compatibility token for already installed openQ4 game modules; pure negotiation cannot download, extract, or restart into executable code, and the token is not cryptographic module attestation. - Added fail-closed read-underflow tracking and range/type checks to audited queued-message, user-command, SP/MP snapshot, server-demo, player, projectile, and hit-scan decode paths. Audited leaf readers stage their fields until valid, and a malformed top-level snapshot tears down the session before another game or presentation frame. diff --git a/docs/dev/savegame-compatibility-policy.md b/docs/dev/savegame-compatibility-policy.md index 1512f3e06..607e0c518 100644 --- a/docs/dev/savegame-compatibility-policy.md +++ b/docs/dev/savegame-compatibility-policy.md @@ -8,11 +8,10 @@ load. Reliability mechanisms and implementation detail are documented in | Save payload | Current decision | | --- | --- | -| Version 3, exact wire-ABI stamp and valid integrity/footer | Supported format path; build/source drift is diagnostic only | -| Version 3, released v0.10 build/source tuple on Windows/MSVC x64 `raw1` | Supported by an exact decoder for its two absent player liquid-state fields and optimized empty physics frame | +| Version 3, exact wire-ABI stamp, build at or above the verified floor, valid integrity/footer | Supported format path; build/source drift is diagnostic only | +| Version 3, build below the verified floor | Rejected before map teardown | | Version 3, different wire-ABI stamp | Rejected before map teardown | -| Version 2, exact tuple in the allowlist below | Supported only on Windows/MSVC x64 little-endian `raw1` | -| Version 2, any other tuple | Rejected before map teardown | +| Version 2, any tuple | Rejected before map teardown | | Unstamped legacy payload | Accepted only when its marker equals the current build and the runtime stamp is Windows/MSVC x64 little-endian `raw1` | | Empty, negative-length, or over-512-MiB `.save` | Rejected before header or CRC preflight | | Payload version newer than 3 | Rejected; no forward-compatibility guessing | @@ -65,28 +64,58 @@ sequence to accept that known omission, while current writers derive class-frame ownership from source declarations so optimization can no longer alter the wire format. -## Exact Version 2 Allowlist +## Version 2 Is No Longer Claimed -Version 2 did not carry its own wire-ABI field. The current reader assigns v2 only -to the known Windows/MSVC x64 little-endian `raw1` lineage and accepts exactly -these `(build, source SHA-256, source-file count)` tuples: +Version 2 did not carry its own wire-ABI field, and support for it was expressed +as an allowlist of `(build, source SHA-256, source-file count)` tuples. On +2026-08-31 every one of those tuples that a real save existed for was tested: +builds 544 (three distinct hashes), 556, and 614. All five desynced part way +through the restore, and did so only after the running map had been torn down -- +the outcome this policy exists to prevent. -| Build | Source SHA-256 | Files | -| ---: | --- | ---: | -| 639 | `d64f5bd29149262e67ce65107ea44b3f10af22011e7af354f23ca01550210fde` | 404 | -| 614 | `0c27fa5c6ef48b1bfe44c7be82b8a696772af4625eeefeed25de27da9640dd3f` | 404 | -| 556 | `871e5811e1732be750b18374b3d537aa38a91a050fb94cef847e2e3d39769cc2` | 218 | -| 544 | `82b545ffb5c9d8d27239eb8d1ed7eb5a22db1c40410dec4f3752f6f90fe76a60` | 218 | -| 544 | `ab567aef25905e8cf52e191523bc591f671b8cee3e63939a67af692bde3de446` | 218 | -| 544 | `9b26849ccdc3652aad892fdeeb5f219b631119fe601de00eb691fb5b4c13e02f` | 218 | +The allowlist was therefore removed rather than corrected. It asserted support +that had never been demonstrated against a real save, and the tuples that were +demonstrable were all wrong. Version 2 payloads are now refused during preflight +with a message naming their provenance. Current writers never create v2 saves. -The three tuple fields plus the running Windows x64 `raw1` ABI are all required -in practice. A matching build alone, hash alone, hash prefix, or file count is -insufficient. Current writers never create new v2 saves. +Restoring a v2 claim requires the same evidence any other claim does: a reviewed +byte-layout comparison, a successful real SP save/load of a save in that exact +format, corruption-contract coverage, and a release-note entry. -An allowlist addition requires a reviewed byte-layout comparison, a successful -real SP save/load on the target Windows x64 runtime, corruption-contract coverage, -and a release-note entry. The allowlist must not be broadened speculatively. +## The Verified Build Floor + +Within version 3, support is expressed as the oldest build whose layout a +verified decoder covers (`SESSION_OPENQ4_SAVEGAME_MINIMUM_SUPPORTED_BUILD`, +currently 661). Payloads below it are refused during preflight. + +661 is where the first of the two player liquid fields entered the save. Builds +from 661 upward restore cleanly, including the range that previously failed; +below it the only sample available, the released v0.10 payload (build 1), +desynced part way through the restore even with its bounded field decoder, so +that lineage is no longer claimed either. + +Lowering the floor requires a real save in the older format that restores +cleanly, not a layout argument alone. + +## Fields Added Within A Version + +A field added to a saved class without a version bump splits the format in two +even though both halves report the same version. Each such field therefore +carries the build that introduced it, and the reader consults that build rather +than a snapshot identity: + +| Field | Added by | Build | +| --- | --- | ---: | +| `idPhysics_Player::swimSpeed` | openQ4-game `2cc5a61`, 2026-08-13 | 661 | +| `idPlayer::nextLiquidSurfaceSoundTime` | openQ4-game `d06a09d`, 2026-08-19 | 721 | + +These two landed six days apart, so a save can legitimately carry the first and +not the second. A single boolean gated both against one hard-coded v0.10 tuple, +which answered "present" for every other v3 payload: every save written between +those builds read a field its file does not contain and desynced the rest of the +restore. Adding a field this way is still discouraged -- a version bump is +clearer -- but when it happens the threshold must be recorded here and in both +game trees. ## Unstamped Legacy Payloads @@ -106,8 +135,8 @@ whole-file CRC/footer retroactively. Backward compatibility means a newer runtime reading an older save. It is supported only through an explicit decoder or allowlist: -- current v3 on the exact ABI path; -- the exact released v0.10 v3 snapshot through its bounded player-field decoder +- current v3 on the exact ABI path at or above the verified build floor; +- fields added within v3, through their recorded per-field build thresholds and legacy empty-physics-frame recognition; - the six exact v2 snapshots above on Windows x64 `raw1`; and - the narrow same-build unstamped Windows x64 legacy path. diff --git a/src/framework/LevelLoadCacheManager.cpp b/src/framework/LevelLoadCacheManager.cpp index c86e5e46d..4a55b2bc8 100644 --- a/src/framework/LevelLoadCacheManager.cpp +++ b/src/framework/LevelLoadCacheManager.cpp @@ -31,6 +31,7 @@ any cached payload can be accepted. #include #include #include +#include #include #include #include @@ -191,29 +192,39 @@ unsigned int DefaultPriority( const levelLoadResourceType_t type ) { } } +bool SemanticTypeUsesCompiledSuffix( const levelLoadResourceType_t type ) { + return type == LEVEL_LOAD_RESOURCE_ANIMATION || + type == LEVEL_LOAD_RESOURCE_RENDER_MODEL || type == LEVEL_LOAD_RESOURCE_WORLD; +} + +bool SemanticTypeMatchesByStem( const levelLoadResourceType_t type ) { + return type == LEVEL_LOAD_RESOURCE_IMAGE || type == LEVEL_LOAD_RESOURCE_SOUND; +} + +std::string SemanticCompiledKey( const std::string &semanticName ) { + std::string compiledName = semanticName; + compiledName += Lexer::sCompiledFileSuffix.c_str(); + return compiledName; +} + +std::string SemanticStemKey( const std::string &path ) { + const std::size_t dot = path.find_last_of( '.' ); + return dot == std::string::npos ? path : path.substr( 0, dot ); +} + bool SemanticPathMatchesSource( const levelLoadResourceType_t type, const std::string &semanticName, const std::string &sourcePath ) { if ( semanticName == sourcePath ) { return true; } - if ( type == LEVEL_LOAD_RESOURCE_ANIMATION || - type == LEVEL_LOAD_RESOURCE_RENDER_MODEL || type == LEVEL_LOAD_RESOURCE_WORLD ) { - std::string compiledName = semanticName; - compiledName += Lexer::sCompiledFileSuffix.c_str(); - if ( compiledName == sourcePath ) { - return true; - } + if ( SemanticTypeUsesCompiledSuffix( type ) && + SemanticCompiledKey( semanticName ) == sourcePath ) { + return true; } - if ( type != LEVEL_LOAD_RESOURCE_IMAGE && type != LEVEL_LOAD_RESOURCE_SOUND ) { + if ( !SemanticTypeMatchesByStem( type ) ) { return false; } - const std::size_t semanticDot = semanticName.find_last_of( '.' ); - const std::size_t sourceDot = sourcePath.find_last_of( '.' ); - const std::string semanticStem = semanticDot == std::string::npos - ? semanticName : semanticName.substr( 0, semanticDot ); - const std::string sourceStem = sourceDot == std::string::npos - ? sourcePath : sourcePath.substr( 0, sourceDot ); - return semanticStem == sourceStem; + return SemanticStemKey( semanticName ) == SemanticStemKey( sourcePath ); } bool BuildSourceIdentity( const std::string &normalizedPath, idFile *file, @@ -533,6 +544,57 @@ struct idLevelLoadCacheManager::Impl { idLevelLoadCache::Manifest learned; idLevelLoadCache::ManifestExpectation expectation; std::vector hints; + // Every opened source used to be matched against the whole hint vector, + // and each recorded hint re-opened its own source, so recording a level's + // media was quadratic in the hint count. These indexes answer the same + // three match rules directly. A key always maps to the newest hint that + // owns it, and lookup takes the highest matching hint index, which is the + // hint the old reverse scan would have stopped on. + std::unordered_map hintExactIndex; + std::unordered_map hintCompiledIndex; + std::unordered_map hintStemIndex; + + // Returns true when this exact semantic name had already been indexed + // during this generation, meaning its source identity is already learned. + bool IndexHint( const std::size_t hintIndex ) { + const SemanticHint &hint = hints[ hintIndex ]; + const bool alreadyIndexed = hintExactIndex.find( hint.normalizedName ) != hintExactIndex.end(); + hintExactIndex[ hint.normalizedName ] = hintIndex; + if ( SemanticTypeUsesCompiledSuffix( hint.type ) ) { + hintCompiledIndex[ SemanticCompiledKey( hint.normalizedName ) ] = hintIndex; + } + if ( SemanticTypeMatchesByStem( hint.type ) ) { + hintStemIndex[ SemanticStemKey( hint.normalizedName ) ] = hintIndex; + } + return alreadyIndexed; + } + + void ClearHints() { + hints.clear(); + hintExactIndex.clear(); + hintCompiledIndex.clear(); + hintStemIndex.clear(); + } + + const SemanticHint *FindHintForSource( const std::string &sourcePath ) const { + bool found = false; + std::size_t best = 0; + const auto consider = [&]( const std::unordered_map &index, + const std::string &key ) { + const auto it = index.find( key ); + if ( it == index.end() ) { + return; + } + if ( !found || it->second > best ) { + found = true; + best = it->second; + } + }; + consider( hintExactIndex, sourcePath ); + consider( hintCompiledIndex, sourcePath ); + consider( hintStemIndex, SemanticStemKey( sourcePath ) ); + return found ? &hints[ best ] : nullptr; + } std::mutex mutex; void ClosePipelineFiles() { @@ -627,7 +689,7 @@ void idLevelLoadCacheManager::Begin( const char *mapKey, const char *gameMode, impl->expectation.entityFilter = impl->learned.entityFilter; impl->expectation.contentSignature = impl->learned.contentSignature; impl->expectation.settingsSignature = impl->learned.settingsSignature; - impl->hints.clear(); + impl->ClearHints(); impl->manifestMatched = false; impl->manifestRemoved = false; impl->generatedHits.store( 0, std::memory_order_relaxed ); @@ -781,7 +843,7 @@ void idLevelLoadCacheManager::Finish( const bool successful ) { impl->completedGeneration = true; impl->recording = true; } else { - impl->hints.clear(); + impl->ClearHints(); impl->learned = idLevelLoadCache::Manifest(); impl->completedGeneration = false; } @@ -814,7 +876,7 @@ void idLevelLoadCacheManager::Cancel() { wroteFinalManifest ? 1 : 0 ); } impl->pipeline.Reset(); - impl->hints.clear(); + impl->ClearHints(); impl->learned = idLevelLoadCache::Manifest(); impl->completedGeneration = false; } @@ -826,6 +888,7 @@ void idLevelLoadCacheManager::RecordSemantic( const levelLoadResourceType_t type if ( impl == nullptr || !NormalizePath( name, normalized ) ) { return; } + bool alreadyResolved = false; { std::lock_guard lock( impl->mutex ); if ( !impl->recording || impl->hints.size() >= 65536 ) { @@ -843,11 +906,19 @@ void idLevelLoadCacheManager::RecordSemantic( const levelLoadResourceType_t type hint.options.assign( options, options + length ); } impl->hints.push_back( std::move( hint ) ); + alreadyResolved = impl->IndexHint( impl->hints.size() - 1 ); } // A semantic lookup may be satisfied by an already resident manager object // and therefore perform no source I/O this generation. Resolve only the // identity (without reading bytes) so those real uses are still learned. + // Media is requested far more often than it is unique, and on a cold file + // cache each speculative open is a real disk touch, so a name that has + // already been resolved this generation is not resolved again: the second + // open would yield the identity the first one already learned. + if ( alreadyResolved ) { + return; + } idFile *source = impl->fileSystem->OpenFileRead( normalized.c_str(), false ); if ( source != nullptr ) { impl->fileSystem->CloseFile( source ); @@ -873,14 +944,11 @@ void idLevelLoadCacheManager::RecordOpenedSource( const char *path, idFile *sour unsigned int priority = DefaultPriority( type ); unsigned int flags = 0; std::vector options; - for ( auto hint = impl->hints.rbegin(); hint != impl->hints.rend(); ++hint ) { - if ( SemanticPathMatchesSource( hint->type, hint->normalizedName, normalized ) ) { - type = hint->type; - priority = hint->priority; - flags = hint->flags; - options = hint->options; - break; - } + if ( const Impl::SemanticHint *hint = impl->FindHintForSource( normalized ) ) { + type = hint->type; + priority = hint->priority; + flags = hint->flags; + options = hint->options; } idLevelLoadCache::ManifestEntry entry; entry.type = ToFormatType( type ); diff --git a/src/framework/Session.cpp b/src/framework/Session.cpp index f0db366e4..859258d4f 100644 --- a/src/framework/Session.cpp +++ b/src/framework/Session.cpp @@ -312,6 +312,11 @@ static const char *SESSION_SAVEGAME_NO_OVERWRITE_TOKEN = "nooverwrite"; static const int SESSION_OPENQ4_SAVEGAME_COMPATIBILITY_MAGIC = 'O' | ( 'Q' << 8 ) | ( '4' << 16 ) | ( 'S' << 24 ); static const int SESSION_OPENQ4_SAVEGAME_COMPATIBILITY_VERSION = 3; static const int SESSION_OPENQ4_SAVEGAME_PREVIOUS_COMPATIBILITY_VERSION = 2; +// Oldest v3 payload this build has a verified decoder for. Saves at or above it +// restore cleanly; the released v0.10 payload (build 1) still desyncs part way +// through, and it did so only after the running map had been torn down. Refuse +// anything below the verified floor during preflight, while the session is intact. +static const int SESSION_OPENQ4_SAVEGAME_MINIMUM_SUPPORTED_BUILD = 661; static const int SESSION_OPENQ4_SAVEGAME_FOOTER_MAGIC = 'O' | ( 'Q' << 8 ) | ( '4' << 16 ) | ( 'F' << 24 ); static const int SESSION_OPENQ4_SAVEGAME_FOOTER_VERSION = 1; static const int SESSION_OPENQ4_SAVEGAME_FOOTER_BYTES = 5 * static_cast( sizeof( int ) ); @@ -322,37 +327,9 @@ static const int SESSION_OPENQ4_SAVEGAME_SOURCE_STAMP_MAX_BYTES = 128; static const int SESSION_OPENQ4_SAVEGAME_ABI_STAMP_MAX_BYTES = 96; static const char *SESSION_LEGACY_SAVEGAME_WIRE_ABI = "windows-msvcabi-x64-le-raw1"; -struct sessionSaveGameV2Snapshot_t { - int build; - const char *sourceHash; - int sourceFileCount; - const char *wireABI; -}; - -static const sessionSaveGameV2Snapshot_t SESSION_OPENQ4_SAVEGAME_V2_SNAPSHOTS[] = { - { 639, "d64f5bd29149262e67ce65107ea44b3f10af22011e7af354f23ca01550210fde", 404, "windows-msvcabi-x64-le-raw1" }, - { 614, "0c27fa5c6ef48b1bfe44c7be82b8a696772af4625eeefeed25de27da9640dd3f", 404, "windows-msvcabi-x64-le-raw1" }, - { 556, "871e5811e1732be750b18374b3d537aa38a91a050fb94cef847e2e3d39769cc2", 218, "windows-msvcabi-x64-le-raw1" }, - { 544, "82b545ffb5c9d8d27239eb8d1ed7eb5a22db1c40410dec4f3752f6f90fe76a60", 218, "windows-msvcabi-x64-le-raw1" }, - { 544, "ab567aef25905e8cf52e191523bc591f671b8cee3e63939a67af692bde3de446", 218, "windows-msvcabi-x64-le-raw1" }, - { 544, "9b26849ccdc3652aad892fdeeb5f219b631119fe601de00eb691fb5b4c13e02f", 218, "windows-msvcabi-x64-le-raw1" } -}; - #ifndef ID_DEDICATED static const char *Session_GetSaveGameWireABI( void ); -static bool Session_IsSupportedSaveGameV2Snapshot( int build, const idStr &sourceHash, int sourceFileCount ) { - for ( int i = 0; i < static_cast( sizeof( SESSION_OPENQ4_SAVEGAME_V2_SNAPSHOTS ) / sizeof( SESSION_OPENQ4_SAVEGAME_V2_SNAPSHOTS[0] ) ); i++ ) { - const sessionSaveGameV2Snapshot_t &snapshot = SESSION_OPENQ4_SAVEGAME_V2_SNAPSHOTS[i]; - if ( build == snapshot.build && sourceFileCount == snapshot.sourceFileCount && - sourceHash.Icmp( snapshot.sourceHash ) == 0 && - idStr::Icmp( Session_GetSaveGameWireABI(), snapshot.wireABI ) == 0 ) { - return true; - } - } - return false; -} - static const char *Session_GetSaveGameWireABI( void ) { #if defined( _WIN32 ) #define SESSION_SAVEGAME_ABI_OS "windows" @@ -896,16 +873,20 @@ static bool Session_ValidateSaveGamePayload( idFile *file, const idStr &savePath } if ( valid && payloadVersion == SESSION_OPENQ4_SAVEGAME_PREVIOUS_COMPATIBILITY_VERSION ) { - if ( !Session_IsSupportedSaveGameV2Snapshot( payloadBuild, payloadSourceStamp, payloadSourceFileCount ) ) { - common->Warning( "Savegame '%s' uses an unsupported v%d source snapshot %s (%d files), build %d", - savePath.c_str(), payloadVersion, payloadSourceStamp.c_str(), payloadSourceFileCount, payloadBuild ); - valid = false; - } + // Every v2 snapshot that was tested against a real save desynced part way + // through the restore, so none of them are claimed any more. + common->Warning( "Savegame '%s' was written by an older openQ4 (v%d payload, build %d) and cannot be restored by this build", + savePath.c_str(), payloadVersion, payloadBuild ); + valid = false; } else if ( valid ) { if ( payloadWireABI.Icmp( Session_GetSaveGameWireABI() ) != 0 ) { common->Warning( "Savegame '%s' wire ABI '%s' is incompatible with this '%s' build", savePath.c_str(), payloadWireABI.c_str(), Session_GetSaveGameWireABI() ); valid = false; + } else if ( payloadBuild < SESSION_OPENQ4_SAVEGAME_MINIMUM_SUPPORTED_BUILD ) { + common->Warning( "Savegame '%s' was written by an older openQ4 (build %d, below build %d, the oldest this build can restore)", + savePath.c_str(), payloadBuild, SESSION_OPENQ4_SAVEGAME_MINIMUM_SUPPORTED_BUILD ); + valid = false; } else if ( payloadBuild != BUILD_NUMBER || OPENQ4_SAVEGAME_COMPAT_SOURCE_FILE_COUNT < 0 || payloadSourceStamp.Icmp( OPENQ4_SAVEGAME_COMPAT_SOURCE_HASH ) != 0 || diff --git a/src/renderer/ImageManager.cpp b/src/renderer/ImageManager.cpp index e9276cb67..f64114d53 100644 --- a/src/renderer/ImageManager.cpp +++ b/src/renderer/ImageManager.cpp @@ -1249,6 +1249,13 @@ int idImageManager::LoadLevelImages( bool pacifier ) { continue; } if ( image->levelLoadReferenced && !image->IsLoaded() ) { + if ( pendingIndex >= pendingImages.Num() ) { + // CountPendingLevelLoads and this loop share a predicate today, so + // this cannot trip. Keep the bound anyway: the array is pre-sized + // from that count, and any future divergence would otherwise be a + // silent heap overflow rather than a dropped image. + break; + } pendingImages[ pendingIndex ].image = image; pendingImages[ pendingIndex ].size = 0; pendingImages[ pendingIndex ].index = i; diff --git a/src/renderer/RenderSystem_init.cpp b/src/renderer/RenderSystem_init.cpp index fdb994a17..d9c15005a 100644 --- a/src/renderer/RenderSystem_init.cpp +++ b/src/renderer/RenderSystem_init.cpp @@ -379,7 +379,7 @@ idCVar r_shadowMapTranslucentMinVariance( "r_shadowMapTranslucentMinVariance", " idCVar r_shadowMapTranslucentBleedReduction( "r_shadowMapTranslucentBleedReduction", "0.0", CVAR_RENDERER | CVAR_ARCHIVE | CVAR_FLOAT, "light-bleed reduction applied to translucent shadow moment resolve", 0.0f, 0.95f ); idCVar r_shadowMapGpuSyncTimings( "r_shadowMapGpuSyncTimings", "0", CVAR_RENDERER | CVAR_BOOL, "diagnostic-only: glFinish around shadow-map passes to report GPU-synchronized milliseconds" ); idCVar r_shadowMapGpuTimerQueries( "r_shadowMapGpuTimerQueries", "1", CVAR_RENDERER | CVAR_BOOL, "use non-blocking GL timer queries for shadow-map GPU diagnostics when available" ); -idCVar r_softParticles( "r_softParticles", "0", CVAR_RENDERER | CVAR_ARCHIVE | CVAR_BOOL, "depth-fade eligible BSE particles against opaque scene depth when GLSL is available" ); +idCVar r_softParticles( "r_softParticles", "1", CVAR_RENDERER | CVAR_ARCHIVE | CVAR_BOOL, "depth-fade eligible BSE particles against opaque scene depth when GLSL is available" ); idCVar r_softParticleFadeDistance( "r_softParticleFadeDistance", "64", CVAR_RENDERER | CVAR_ARCHIVE | CVAR_FLOAT, "world-unit distance over which r_softParticles fades particle intersections", 1.0f, 512.0f ); idCVar r_enhancedMaterials( "r_enhancedMaterials", "0", CVAR_RENDERER | CVAR_ARCHIVE | CVAR_BOOL, "use enhanced GLSL interaction shading for existing materials when supported" ); idCVar r_enhancedMaterialNormalScale( "r_enhancedMaterialNormalScale", "1.25", CVAR_RENDERER | CVAR_ARCHIVE | CVAR_FLOAT, "tangent-space normal XY scale when enhanced material shading is enabled", 0.5f, 2.0f ); diff --git a/src/renderer/RenderWorld_load.cpp b/src/renderer/RenderWorld_load.cpp index 6d9574f74..5733a7f34 100644 --- a/src/renderer/RenderWorld_load.cpp +++ b/src/renderer/RenderWorld_load.cpp @@ -462,6 +462,8 @@ namespace { static const int RENDER_WORLD_CACHE_MAX_MODELS = 65536; static const int RENDER_WORLD_CACHE_MAX_MODEL_PAYLOAD = 128 * 1024 * 1024; static const uint64_t RENDER_WORLD_CACHE_MAX_MODEL_MEMORY = 384ULL * 1024ULL * 1024ULL; + static const int RENDER_WORLD_CACHE_MAX_SURFACE_VERTS = 1 << 20; + static const int RENDER_WORLD_CACHE_MAX_SURFACE_INDEXES = 1 << 24; static const int RENDER_WORLD_CACHE_MAX_SHADOW_VERTS = 1 << 20; static const int RENDER_WORLD_CACHE_MAX_SHADOW_INDEXES = 1 << 24; static const int RENDER_WORLD_CACHE_MAX_AREAS = 4096; @@ -1625,6 +1627,17 @@ idRenderModel *idRenderWorldLocal::ParseModel( Lexer *src ) { tri->numVerts = src->ParseInt(); tri->numIndexes = src->ParseInt(); + // FinishSurfaces below dereferences every index while deriving tangents + // and silhouette edges, so the file-provided counts have to be sane + // before anything is allocated. ParseShadowModel and the binary + // render-world cache already apply the same predicates. + if ( tri->numVerts < 0 || tri->numVerts > RENDER_WORLD_CACHE_MAX_SURFACE_VERTS + || tri->numIndexes < 0 || tri->numIndexes > RENDER_WORLD_CACHE_MAX_SURFACE_INDEXES + || ( tri->numIndexes % 3 ) != 0 ) { + src->Error( "R_ParseModel: bad surface counts" ); + return NULL; + } + R_AllocStaticTriSurfVerts( tri, tri->numVerts ); for ( j = 0 ; j < tri->numVerts ; j++ ) { // jmarshall - quake 4 proc format @@ -1664,7 +1677,12 @@ idRenderModel *idRenderWorldLocal::ParseModel( Lexer *src ) { R_AllocStaticTriSurfIndexes( tri, tri->numIndexes ); for ( j = 0 ; j < tri->numIndexes ; j++ ) { - tri->indexes[j] = src->ParseInt(); + const int index = src->ParseInt(); + if ( index < 0 || index >= tri->numVerts ) { + src->Error( "R_ParseModel: index %i out of range (%i verts)", index, tri->numVerts ); + return NULL; + } + tri->indexes[j] = index; } src->ExpectTokenString( "}" ); diff --git a/src/ui/DeviceContext.cpp b/src/ui/DeviceContext.cpp index 70ed031b1..d1fe5728c 100644 --- a/src/ui/DeviceContext.cpp +++ b/src/ui/DeviceContext.cpp @@ -793,6 +793,24 @@ static float openQ4_FontSelectionScaleForViewport( float authoredScale, float ca return authoredScale * Max( 1.0f, physicalScale ); } +// A scalable font rasterises its small/medium/large slots from one face at +// 12/24/48 point, so their point-size-normalised metrics agree and picking a +// larger slot only buys resolution. The retail .fontdat atlases were authored +// by hand per size and do not agree to the pixel, so picking a different one +// there moves text. Only the scalable slots carry the "ttf/" name prefix. +static bool openQ4_FontAtlasesAreProportional( const fontInfoEx_t *font ) { + if ( font == NULL ) { + return false; + } + const fontInfo_t *slots[3] = { &font->fontInfoSmall, &font->fontInfoMedium, &font->fontInfoLarge }; + for ( int i = 0; i < 3; i++ ) { + if ( idStr::Cmpn( slots[i]->name, "ttf/", 4 ) != 0 ) { + return false; + } + } + return true; +} + static float openQ4_FontSelectionScale( float authoredScale, float canvasWidth, float canvasHeight, bool aspectCorrect ) { float viewportWidth = 0.0f; float viewportHeight = 0.0f; @@ -2226,11 +2244,18 @@ void idDeviceContext::SetFontByScale(float scale) { return; } // The GUI scale is authored for a 640x480-era canvas, while the selected - // bitmap atlas is sampled at the final viewport resolution. Account for - // that physical enlargement so high-resolution displays use the 24/48-point - // source instead of magnifying a small atlas. Rendering still uses the - // authored scale below, so text layout and dimensions do not change. - const float selectionScale = openQ4_FontSelectionScale( scale, vidWidth, vidHeight, aspectCorrect ); + // atlas is sampled at the final viewport resolution. Account for that + // physical enlargement so high-resolution displays use the 24/48-point + // source instead of magnifying a small atlas. + // + // useFont is also what TextWidth, MaxCharHeight and the DrawText advances + // read, so this may only move between atlases whose normalised metrics + // agree. Scalable slots qualify because one face produced all three; the + // retail .fontdat atlases do not, and moving between those would shift + // text width, line height and right alignment on a large window. + const float selectionScale = openQ4_FontAtlasesAreProportional( activeFont ) + ? openQ4_FontSelectionScale( scale, vidWidth, vidHeight, aspectCorrect ) + : scale; if (selectionScale <= gui_smallFontLimit.GetFloat()) { useFont = &activeFont->fontInfoSmall; activeFont->maxHeight = activeFont->maxHeightSmall; @@ -3086,6 +3111,30 @@ bool UI_FontParity_RunSelfTest( void ) { ok &= openQ4_CheckNear( "1440p font selection scale", openQ4_FontSelectionScaleForViewport( 0.25f, 640.0f, 480.0f, 2560.0f, 1440.0f, true ), 0.75f ); + // The enlargement above may only choose between atlases whose normalised + // metrics agree, because the chosen atlas is also what TextWidth, + // MaxCharHeight and the DrawText advances read. Scalable slots qualify; + // the hand-authored retail atlases do not. + fontInfoEx_t scalableAtlasFont = {}; + idStr::Copynz( scalableAtlasFont.fontInfoSmall.name, "ttf/marine_12", sizeof( scalableAtlasFont.fontInfoSmall.name ) ); + idStr::Copynz( scalableAtlasFont.fontInfoMedium.name, "ttf/marine_24", sizeof( scalableAtlasFont.fontInfoMedium.name ) ); + idStr::Copynz( scalableAtlasFont.fontInfoLarge.name, "ttf/marine_48", sizeof( scalableAtlasFont.fontInfoLarge.name ) ); + ok &= openQ4_CheckBool( "scalable atlases are proportional", + openQ4_FontAtlasesAreProportional( &scalableAtlasFont ), true ); + + fontInfoEx_t retailAtlasFont = {}; + idStr::Copynz( retailAtlasFont.fontInfoSmall.name, "fonts/english/marine_12.fontdat", sizeof( retailAtlasFont.fontInfoSmall.name ) ); + idStr::Copynz( retailAtlasFont.fontInfoMedium.name, "fonts/english/marine_24.fontdat", sizeof( retailAtlasFont.fontInfoMedium.name ) ); + idStr::Copynz( retailAtlasFont.fontInfoLarge.name, "fonts/english/marine_48.fontdat", sizeof( retailAtlasFont.fontInfoLarge.name ) ); + ok &= openQ4_CheckBool( "retail atlases are not proportional", + openQ4_FontAtlasesAreProportional( &retailAtlasFont ), false ); + + // A font that produced only some scalable slots must not qualify either. + fontInfoEx_t mixedAtlasFont = scalableAtlasFont; + idStr::Copynz( mixedAtlasFont.fontInfoLarge.name, "fonts/english/marine_48.fontdat", sizeof( mixedAtlasFont.fontInfoLarge.name ) ); + ok &= openQ4_CheckBool( "mixed atlases are not proportional", + openQ4_FontAtlasesAreProportional( &mixedAtlasFont ), false ); + glyphInfo_t glyph = {}; glyph.horiAdvance = 7.2f; glyph.height = 11.9f; diff --git a/tools/tests/savegame_corruption_contract.py b/tools/tests/savegame_corruption_contract.py index 4179839da..59fe1097a 100644 --- a/tools/tests/savegame_corruption_contract.py +++ b/tools/tests/savegame_corruption_contract.py @@ -475,7 +475,7 @@ def validate_session_source_contract() -> None: "SESSION_OPENQ4_SAVEGAME_INTEGRITY_BYTES", "SESSION_OPENQ4_SAVEGAME_COMPATIBILITY_VERSION = 3", "SESSION_OPENQ4_SAVEGAME_PREVIOUS_COMPATIBILITY_VERSION = 2", - "static bool Session_IsSupportedSaveGameV2Snapshot( int build, const idStr &sourceHash, int sourceFileCount )", + "SESSION_OPENQ4_SAVEGAME_MINIMUM_SUPPORTED_BUILD", "static const char *Session_GetSaveGameWireABI( void )", "SESSION_MAX_SAVE_DESCRIPTION_BYTES = 8192", "SESSION_MAX_SAVE_PREVIEW_BYTES = 64 * 1024 * 1024", @@ -487,7 +487,7 @@ def validate_session_source_contract() -> None: "OPENQ4_SAVEGAME_COMPAT_SOURCE_HASH", "OPENQ4_SAVEGAME_COMPAT_SOURCE_FILE_COUNT", "if ( marker != SESSION_OPENQ4_SAVEGAME_COMPATIBILITY_MAGIC )", - "!Session_IsSupportedSaveGameV2Snapshot( payloadBuild, payloadSourceStamp, payloadSourceFileCount )", + "payloadBuild < SESSION_OPENQ4_SAVEGAME_MINIMUM_SUPPORTED_BUILD", "payloadWireABI.Icmp( Session_GetSaveGameWireABI() ) != 0", "static bool Session_CalculateSaveGameChecksum( idFile *file, int protectedLength, unsigned int &checksum, const idStr &savePath )", "static bool Session_AppendSaveGameIntegrityTrailer( const idStr &relativePath )", @@ -735,8 +735,9 @@ def validate_gamelibs_save_payload_contract() -> None: "ReadString( openQ4SaveGameCompatibilityStamp );", "ReadInt( openQ4SaveGameCompatibilitySourceFileCount );", "openQ4SaveGameCompatibilityVersion != OPENQ4_SAVEGAME_COMPATIBILITY_VERSION", - "static bool SaveGame_IsSupportedV2Snapshot( int build, const idStr &sourceHash, int sourceFileCount )", - "!SaveGame_IsSupportedV2Snapshot( buildNumber, openQ4SaveGameCompatibilityStamp, openQ4SaveGameCompatibilitySourceFileCount )", + "OPENQ4_SAVEGAME_BUILD_WITH_PLAYER_SWIM_SPEED", + "OPENQ4_SAVEGAME_BUILD_WITH_PLAYER_LIQUID_SOUND", + "has no verified decoder", "ReadString( savedWireABI );", "savedWireABI.Icmp( OpenQ4SaveGameWireABI() ) != 0", "Schema-compatible save source differs", diff --git a/tools/tests/savegame_v3_contract.py b/tools/tests/savegame_v3_contract.py index d40dc86d8..e81dd4ff1 100644 --- a/tools/tests/savegame_v3_contract.py +++ b/tools/tests/savegame_v3_contract.py @@ -17,12 +17,6 @@ INTEGRITY_TRAILER_BYTES = 16 MENU_GUI_ASPECT = 640.0 / 480.0 MENU_PREVIEW_BOUNDS = (25.0, 78.0, 183.0, 137.0) -V3_PRE_PLAYER_LIQUID_FIELDS_SNAPSHOT = ( - 1, - "19351be39d2d4077a74294c0442707ef9565fc7a2fa9af9b81e05fc9aca8b220", - 404, - "windows-msvcabi-x64-le-raw1", -) def read(path: Path) -> str: @@ -300,38 +294,68 @@ def validate_source_contracts() -> None: if constant(source, "OPENQ4_SAVEGAME_PREVIOUS_COMPATIBILITY_VERSION") != 2: raise AssertionError(f"{context} previous save reader differs from engine v2") - engine_snapshots = snapshot_tuples(session, "SESSION_OPENQ4_SAVEGAME_V2_SNAPSHOTS") - sp_snapshots = snapshot_tuples(sp, "OPENQ4_SAVEGAME_V2_SNAPSHOTS") - mp_snapshots = snapshot_tuples(mp, "OPENQ4_SAVEGAME_V2_SNAPSHOTS") - if engine_snapshots != sp_snapshots or engine_snapshots != mp_snapshots: - raise AssertionError("Engine/SP/MP v2 compatibility allowlists differ") - if any(wire_abi != "windows-msvcabi-x64-le-raw1" for _, _, _, wire_abi in engine_snapshots): - raise AssertionError("Ambiguous unstamped v2 snapshots must stay restricted to their known wire ABI") + # A save is only claimed when a verified decoder exists for its layout. The + # per-snapshot allowlists made that claim for tuples that were never + # exercised against a real save, and every one that was later tested + # desynced part way through the restore - after the running map had already + # been torn down. They are replaced by a build floor plus per-field build + # thresholds, so nothing may reintroduce a tuple allowlist. + for source, context in ( + (session, "engine"), + (sp, "SP GameLib"), + (mp, "MP GameLib"), + ): + for banned in ( + "SAVEGAME_V2_SNAPSHOTS", + "V3_PRE_PLAYER_LIQUID_FIELDS_SNAPSHOTS", + "IsSupportedV2Snapshot", + "IsV3PrePlayerLiquidFieldsSnapshot", + ): + if banned in source: + raise AssertionError( + f"{context} still carries the {banned} compatibility allowlist; " + "save support must come from a verified decoder, not a tuple list" + ) - sp_pre_liquid_snapshots = snapshot_tuples( - sp, "OPENQ4_SAVEGAME_V3_PRE_PLAYER_LIQUID_FIELDS_SNAPSHOTS" - ) - mp_pre_liquid_snapshots = snapshot_tuples( - mp, "OPENQ4_SAVEGAME_V3_PRE_PLAYER_LIQUID_FIELDS_SNAPSHOTS" - ) - expected_pre_liquid_snapshots = [V3_PRE_PLAYER_LIQUID_FIELDS_SNAPSHOT] - if sp_pre_liquid_snapshots != expected_pre_liquid_snapshots or mp_pre_liquid_snapshots != expected_pre_liquid_snapshots: - raise AssertionError("SP/MP v0.10 player-liquid compatibility snapshots differ from the approved tuple") + minimum_build = constant(session, "SESSION_OPENQ4_SAVEGAME_MINIMUM_SUPPORTED_BUILD") + if minimum_build <= 0: + raise AssertionError("Engine must declare the oldest v3 build it can restore") + + # The two player liquid fields landed in different builds, so they need + # independent thresholds; one boolean for both is what made saves written + # between them read a field their file does not contain. + sp_swim = constant(sp, "OPENQ4_SAVEGAME_BUILD_WITH_PLAYER_SWIM_SPEED") + mp_swim = constant(mp, "OPENQ4_SAVEGAME_BUILD_WITH_PLAYER_SWIM_SPEED") + sp_sound = constant(sp, "OPENQ4_SAVEGAME_BUILD_WITH_PLAYER_LIQUID_SOUND") + mp_sound = constant(mp, "OPENQ4_SAVEGAME_BUILD_WITH_PLAYER_LIQUID_SOUND") + if sp_swim != mp_swim or sp_sound != mp_sound: + raise AssertionError("SP/MP player liquid field build thresholds differ") + if sp_swim >= sp_sound: + raise AssertionError("swimSpeed was added before nextLiquidSurfaceSoundTime; thresholds must reflect that") + if minimum_build > sp_swim: + raise AssertionError( + "The verified build floor must not exclude saves the liquid thresholds still describe" + ) for source, header, class_header, player, physics_player, context in ( (sp, sp_h, sp_class_h, sp_player, sp_physics_player, "SP GameLib"), (mp, mp_h, mp_class_h, mp_player, mp_physics_player, "MP GameLib"), ): for token in ( - "SaveGame_IsV3PrePlayerLiquidFieldsSnapshot", - "bool idRestoreGame::HasOpenQ4PlayerLiquidSaveFields", + "bool idRestoreGame::HasOpenQ4PlayerSwimSpeedSaveField", + "bool idRestoreGame::HasOpenQ4PlayerLiquidSoundSaveField", + "bool idRestoreGame::HasOpenQ4PlayerLiquidSaveFieldForBuild", "HasNextSerializedEmptyClassFrame", 'idStr::Icmp( cls->classname, "idPhysics" ) == 0', "!cls->saveDeclaredHere", "!cls->restoreDeclaredHere", ): require(source, token, f"{context} v0.10 compatibility decoder") - require(header, "HasOpenQ4PlayerLiquidSaveFields( void ) const", f"{context} compatibility accessor") + for accessor in ( + "HasOpenQ4PlayerSwimSpeedSaveField( void ) const", + "HasOpenQ4PlayerLiquidSoundSaveField( void ) const", + ): + require(header, accessor, f"{context} compatibility accessor") for token in ( "struct idMemberFunctionOwner", "struct idMemberFunctionDeclaredHere", @@ -350,23 +374,23 @@ def validate_source_contracts() -> None: if re.search( r"ReadInt\s*\(\s*previousWaterType\s*\)\s*;\s*" r"nextLiquidSurfaceSoundTime\s*=\s*0\s*;\s*" - r"if\s*\(\s*savefile->HasOpenQ4PlayerLiquidSaveFields\s*\(\s*\)\s*\)\s*\{\s*" + r"if\s*\(\s*savefile->HasOpenQ4PlayerLiquidSoundSaveField\s*\(\s*\)\s*\)\s*\{\s*" r"savefile->ReadInt\s*\(\s*nextLiquidSurfaceSoundTime\s*\)\s*;\s*\}\s*" r"savefile->ReadInt\s*\(\s*nextLiquidDamageTime\s*\)", player, re.DOTALL, ) is None: - raise AssertionError(f"{context} does not restore the pre-v0.10 liquid-sound timer layout") + raise AssertionError(f"{context} does not gate the liquid-sound timer on the build that added it") if re.search( r"ReadFloat\s*\(\s*playerSpeed\s*\)\s*;\s*" r"swimSpeed\s*=\s*0\.0f\s*;\s*" - r"if\s*\(\s*savefile->HasOpenQ4PlayerLiquidSaveFields\s*\(\s*\)\s*\)\s*\{\s*" + r"if\s*\(\s*savefile->HasOpenQ4PlayerSwimSpeedSaveField\s*\(\s*\)\s*\)\s*\{\s*" r"savefile->ReadFloat\s*\(\s*swimSpeed\s*\)\s*;\s*\}\s*" r"savefile->ReadVec3\s*\(\s*viewForward\s*\)", physics_player, re.DOTALL, ) is None: - raise AssertionError(f"{context} does not restore the pre-v0.10 swim-speed layout") + raise AssertionError(f"{context} does not gate swim speed on the build that added it") require(session, 'SESSION_LEGACY_SAVEGAME_WIRE_ABI = "windows-msvcabi-x64-le-raw1"', "engine unstamped legacy ABI restriction")