Repository navigation
drm/asahi: g15: renumber the TA/FS and compute stamps before they wrap - #287
Conversation
The TA stamp of render ordinal n is stamp_seed(TA_SEED, n): it steps by 0x100 and wraps after 0x860000 renders, about six hours of desktop. With the stamp_seed() from before commit 7ccdfe8 ("drm/asahi: wrap G15 stamps through zero"), which skipped zero and went from 0xffffff00 back to 0x100 there, the firmware stopped at render 8781824 with "DAG: Non Sequential Stamp Updates seen ... stampValue 0x100" and the GPU was lost until reboot. Since that commit the stamps wrap through zero; whether the firmware takes that wrap has not been tried. Add an offset to the render ordinal used for the TA/FS stamps and, when g15_stamp_reseed_margin is set, renumber both stamps from their seeds at the first publication with nothing in flight inside that margin: write the seeds to the user and firmware stamps, reset the engine targets and move the offset so the next render counts as ordinal 1, as after boot. The offset is taken modulo the 2^24 ordinals after which stamp_seed() repeats (STAMP_PERIOD). The stamps then never reach the wrap. g15_stamp_base starts the count near the wrap, to test this in seconds instead of hours. Both parameters default to 0, which keeps the stamps exactly as before. Tested on a J516C, together with the next commit, on integration/aurora with the earlier stamp_seed() and the offset taken modulo its period of 0xffffff ordinals: from g15_stamp_base 2000 renders before the wrap with a margin of 1000, the stamps were renumbered at render 1000 and 3003 renders retired; in desktop use, one boot renumbered at renders 8681824 and 17363648. On this base, modulo 0xffffff would give the render after the first renumbering ordinal 0, whose targets the seeds already meet, and the one after the second ordinal -1, a TA target below the seed. Generated-by: LLM Signed-off-by: Samuel Carreño <soyminero11yt@gmail.com>
Renumbering only happens at a publication with nothing in flight. A client that keeps the GPU busy (no vsync) can keep a flight in the air through the whole margin, and the TA stamp would then wrap as before. Inside the margin, publish nothing behind a flight, so the flights drain and the next publication renumbers while idle (once every ~8.7M renders with a margin of 100000). Also clamp a non-zero margin to at least one prefix of renders, so that the last prefix published outside it cannot cross the wrap, and to at most half the stamp range, so that it cannot renumber on every publication. Found while reviewing the previous commit, with a model of prefixes of 1-8 renders: a fully overlapped client wrapped 199/199 times without this and 0/199 with it. On a J516C, with the earlier stamp_seed() (see the previous commit), from g15_stamp_base 500000 renders before the wrap with a margin of 100000 and the GPU kept busy (286875 publications behind a flight), the stamps were renumbered at render 400000 and rendering went on past the wrap point to render 526113. Generated-by: LLM Signed-off-by: Samuel Carreño <soyminero11yt@gmail.com>
The compute stamp of ordinal n is stamp_seed(COMPUTE_SEED, n): it steps by 0x100 from 0xc1000000 and, after 0x3f0000 computes (4128768: about 390 FLUX.3 1024x1024 images at one node per submission), goes from 0xffffff00 back to 0. Whether the firmware takes that wrap has not been tried either; the TA/FS renumbering of the previous commits does not cover it. Renumber the compute stamp the same way. Add an offset to the compute ordinal used for the stamp and, when g15_compute_reseed_margin is set, at the first publication with nothing in flight inside that margin write the compute stamps back as ComputeGraph::init() leaves them with no stamp base (user COMPUTE_SEED, firmware 0), reset the compute target and move the offset so the next compute counts as ordinal 1. Inside the margin nothing is published behind a flight, as for TA/FS, so the flights drain; the margin is at least one prefix of computes (COMPUTE_SLOTS) and at most half the stamp range. The ordinal itself, which the compute image also carries, keeps counting as the render draw does. g15_compute_stamp_base starts the count near the wrap, to test this in seconds instead of hours. Both parameters default to 0, which keeps the stamp exactly as before. On a J516C, with g15_compute_stamp_base=4023768 (105000 computes before the wrap) and g15_compute_reseed_margin=100000, the stamp was renumbered at compute 5000, the GPU kept running computes up to 159899, past the point where the stamp would have wrapped, and a FLUX.3 1024x1024 image generated fine afterwards. Generated-by: LLM Signed-off-by: Samuel Carreño <soyminero11yt@gmail.com>
Add a "stamps:" line to asahi-g15/stats with the computes and renders the current numbering has left before the compute and TA stamps wrap, and both renumbering margins (0: off). A script that limits compute work before the wrap can then tell a kernel that renumbers the compute stamp from one that does not, and count from the stamp rather than from boot (g15_compute_stamp_base moves the start). Generated-by: LLM Signed-off-by: Samuel Carreño <soyminero11yt@gmail.com>
chriskearney
left a comment
There was a problem hiding this comment.
Reviewed all six changed files at exact head c3fad65647f0e46223b91fb7a562d2f1a7c3e945, the drain/retirement and failure paths, and the independent exact-method host controls (26M renders and 10M computes, two renumberings each, default/boundary/write-failure cases and five caught semantic mutants). No blocking source finding for landing this as a default-off experimental facility: all four new parameters default to zero, preserving current stamp generation and publication policy.
This approval does not qualify enabled firmware reseeding or a new board/firmware cohort. The current modulo-2^24 TA/FS path has not reached the wrap boundary on hardware; older TA/FS and compute observations are on different/local stacks. Before calling enabled behavior qualified or making it default-on, run a bounded near-wrap qualification on an already-supported/admitted G15 stack with the existing stop/restore route. A wrap-to-zero crash control or new Max/firmware admission is not required.
Current required CI and full merge-group CI remain required. The C compile job skips Rust; the actual queue-tree package build must retain enabled RUST/DRM_ASAHI/G15 config and successful aarch64 Asahi module build evidence. Normal full packaging can provide that native build; an earlier package run is evidence of the mechanism, not a build of this head.
31748f0
Problem. The G15 stamps step by 0x100 per job from fixed seeds, so they wrap after a fixed count since the driver loaded: the TA stamp after 0x860000 renders (about six hours of desktop), the compute stamp after 0x3f0000 computes (4,128,768: about 390 image generations with ggml-vulkan at one node per submission). On a J516C, with the
stamp_seed()from before 7ccdfe8 (which skipped zero), the TA stamp wrapped at render 8,781,824 and the firmware stopped withDAG: Non Sequential Stamp Updates seen ... stampValue 0x100; the GPU was lost until reboot. 7ccdfe8 now wraps the stamps through zero; I have not tried whether the firmware takes that wrap.Change. Renumber the stamps from their seeds while the GPU is idle, shortly before they would wrap, so they never reach the wrap whatever the firmware does there. Four commits:
g15_stamp_reseed_marginrenders of the TA wrap, write the seeds to the user and firmware stamps, reset the engine targets and move an offset, modulo the 2^24 period ofstamp_seed(), so the next render is ordinal 1, as after boot.g15_compute_reseed_margin: the stamps go back to whatComputeGraph::init()writes with no stamp base (userCOMPUTE_SEED, firmware 0).stamps:line inasahi-g15/stats: computes and renders left before each wrap, and both margins.g15_stamp_baseandg15_compute_stamp_base(testing only) start the counts near the wraps, so this can be tested in seconds. Independent of #261: it applies to integration/aurora as it is and merges cleanly with it and with #258.Enabling it. Every new parameter defaults to 0, and with both margins at 0 nothing changes: no offset, no hold, the same stamps as now. I run both margins at 100000:
(or
asahi.g15_stamp_reseed_margin=100000 asahi.g15_compute_reseed_margin=100000on the kernel command line). They are read when the module loads;asahi-g15/statsshows them (render_margin,compute_margin) with what is left, and the kernel logsG15: TA/FS stamps renumbered ...orG15: compute stamp renumbered ...when it happens, with these margins about once every 8.7M renders or 4.0M computes.Checked on a MacBook Pro 16" M3 Max (J516C, 64 GB), always on top of local patches, among them HDMI on dcpext0 and J516C panel brightness:
free_guiltyfix of Fix the G17 late-completion Oops in drm_sched_job_done() aurora-silicon/linux#381 and an AFK logging change: fromg15_compute_stamp_base=4023768(105,000 computes before the wrap) with a margin of 100000,G15: compute stamp renumbered from the seed at compute 5000 (ordinal 0x3d7960); compute work (test-backend-ops perf -o ADD, one node per submission) went on to 159,899 computes, withcompute_left 3973869at the end, and a FLUX.3 1024x1024 image generated fine afterwards. The same on 30a6518 (159,890 computes).stamp_seed()there. On 8b88dfc: fromg15_stamp_base=8779824with a margin of 1000, renumbered at render 1000 and 3,003 renders retired; fromg15_stamp_base=8281824with a margin of 100000 and the GPU kept busy (286,875 publications behind a flight), renumbered at render 400,000 and went on to 526,113, past render 500,000, where the TA stamp would have wrapped. On 937b727, also with AVD AV1 and drm/asahi: g15: allow 8 prefixes in flight per VM (was 2), tunable with g15_max_inflight #261, in desktop use: one boot renumbered at renders 8,681,824 and 17,363,648. NoNon Sequential, firmware crash or GPU fault in any of them.drivers/gpu/drm/asahi/asahi.obuilds on 2f72554 at every commit (arm64, clang 21, rustc 1.98; the CI compile check does not build Rust), with no new warnings; the G15 host selftests pass.Not checked. Whether the firmware takes the wrap through zero: with a margin of 0 and a stamp base near the wrap this series makes it a test of seconds, but a stop costs a reboot and I have not run it. Other boards; this branch alone on hardware.
Model of the TA targets
🤖 Generated with Claude Code