[GSD-13279] Silent wrong results on Arc A770 (DG2): reused per-dispatch private surface never made resident again — evictUnusedAllocations() unbinds it permanently (regression since ee21f7c717)

[AIVirtuoso]

Status 2026-09-02 — the fix is on master, not yet in a release. 3d7a21dca9 "fix: add reused private memory to residency container" (2026-08-27, Related-To: GSD-13279) makes exactly the source change proposed below — byte for byte the patch I built and A/B-tested on this hardware — and adds four unit tests, including the immediate-command-list case this report is about. As of today it is not in any release (newest: 26.31.39395.13, 2026-08-21) and not on releases/26.31 or releases/26.35. Three requests: (1) confirm here that 3d7a21dca9 is the fix for this issue; (2) backport it to the release branches — this is a silent wrong-results regression, PyTorch on XPU hits it through its cat/stack kernels, and any SYCL program with a large enough module and file-backed input is exposed; (3) keep the issue open until a release carries it. The report below is kept current for whoever reviews or backports the change: it names the failing line, the exact chain from that line to wrong results, and — new since the original filing — pins the exposure to immediate command lists by measurement.

Pre-submission Checklist

• I am using the latest GPU driver version (releases)
• I have searched for similar issues and found none

GPU Hardware

Intel Arc A770 16 GB (DG2-512 / ACM-G10), SPARKLE A770 TITAN OC Edition (SA770T-16GOC), single discrete GPU, no integrated GPU in use.

DRI Devices Information

$ ls -ls /dev/dri/*
0 crw-rw----+ 1 root video  226,   0 ago 10 13:16 /dev/dri/card0
0 crw-rw-rw-  1 root render 226, 128 ago 10 13:15 /dev/dri/renderD128

$ ls -la /dev/dri/by-path/
lrwxrwxrwx 1 root root  8 ago 10 13:15 pci-0000:09:00.0-card -> ../card0
lrwxrwxrwx 1 root root  8 ago 10 13:14 pci-0000:09:00.0-platform-simple-framebuffer.0-card -> ../card0
lrwxrwxrwx 1 root root 13 ago 10 13:15 pci-0000:09:00.0-render -> ../renderD128

GPU Detailed Information (lspci output)

$ sudo lspci -vvv -k -s 0000:09:00.0
09:00.0 VGA compatible controller: Intel Corporation DG2 [Arc A770] (rev 08) (prog-if 00 [VGA controller])
	Subsystem: Sparkle Computer Co., Ltd. Device 3937
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupts: unknown pin routed to IRQ 77, MSI(X) routed to IRQ 77
	IOMMU group: 19
	Region 0: Memory at fb000000 (64-bit, non-prefetchable) [size=16M]
	Region 2: Memory at 7800000000 (64-bit, prefetchable) [size=16G]
	Expansion ROM at fc000000 [disabled] [size=2M]
	Capabilities: [40] Vendor Specific Information: Intel Capabilities v1
		CapA: Peg60Dis- Peg12Dis- Peg11Dis- Peg10Dis- PeLWUDis- DmiWidth=x4
		      EccDis- ForceEccEn- VTdDis- DmiG2Dis- PegG2Dis- DDRMaxSize=Unlimited
		      1NDis- CDDis- DDPCDis- X2APICEn- PDCDis- IGDis- CDID=0 CRID=0
		      DDROCCAP- OCEn- DDRWrtVrefEn+ DDR3LEn+
		CapB: ImguDis- OCbySSKUCap- OCbySSKUEn- SMTCap- CacheSzCap 0x0
		      SoftBinCap- DDR3MaxFreqWithRef100=Disabled PegG3Dis-
		      PkgTyp- AddGfxEn- AddGfxCap- PegX16Dis- DmiG3Dis- GmmDis-
		      DDR3MaxFreq=2932MHz LPDDR3En-
	Capabilities: [70] Express (v2) Endpoint, IntMsgNum 0
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W TEE-IO-
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- FltModeDis-
		LnkSta:	Speed 2.5GT/s, Width x1
			TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
			 10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- TPHComp- ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
			 AtomicOpsCtl: ReqEn-
			 IDOReq- IDOCompl- LTR+ EmergencyPowerReductionReq-
			 10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
		LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported, FltMode-
	Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
		Address: 00000000fee00000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [d0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [420 v1] Physical Resizable BAR
		BAR 2: current size: 16GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB
	Capabilities: [400 v1] Latency Tolerance Reporting
		Max snoop latency: 1048576ns
		Max no snoop latency: 1048576ns
	Kernel driver in use: xe
	Kernel modules: i915, xe

Resizable BAR is enabled (BAR 2 at the full 16 GB). Note the link reads Speed 2.5GT/s, Width x1 — that is what lspci reports with the GPU idle in ASPM L1; under load the reproducer sustains 1.3-2.5 GB/s host-to-device, an order of magnitude more than a real 2.5GT/s x1 link could carry, so the link is not actually running at x1 when it matters.

Driver Version

26.18.38308.1

Installed GPU Driver Packages

NixOS, so there is no dpkg/rpm database to query. The relevant store paths:

intel-compute-runtime-26.18.38308.1
level-zero-1.28.5
intel-graphics-compiler-2.34.4
intel-gmmlib-22.10.0

The Level Zero device itself reports driver_version 1.15.38308, i.e. the same 26.18.38308.1 build named in the Driver Version field.

The Level Zero loader in use is the system libze_loader.so.1 (1.28.5); the SYCL runtime and UR adapters come from the pip intel-sycl-rt / intel-cmplr-lib-ur packages inside a Python venv, versions given under oneAPI Version below.

Driver Installation Details

• Installation method: NixOS 26.05 declarative configuration (hardware.graphics.extraPackages pulling intel-compute-runtime from nixpkgs), not the Intel apt repository.
• Kernel driver: xe, forced on this device via boot parameters i915.force_probe=!56a0 xe.force_probe=56a0 (both modules are present; i915 is loaded but bound to nothing). i915 is untested — see "Still open", item 1, in Additional Notes.
• The application runs inside an FHS-compatible shell so that a normal Python venv with the pip oneAPI runtimes works on NixOS.

Linux Distribution

Other (please specify below)

Other Linux Distribution

NixOS 26.05 (Yarara), BUILD_ID=26.05.20260814.02e0898

Kernel Version & Boot Parameters

$ uname -r
7.1.7

$ cat /proc/cmdline
initrd=\EFI\nixos\...-initrd-linux-7.1.7-initrd.efi init=/nix/store/...-nixos-system.../init
i915.force_probe=!56a0 xe.force_probe=56a0 root=fstab loglevel=4
video=HDMI-A-4:e drm.edid_firmware=HDMI-A-4:edid/SAM_1080p.bin
lsm=landlock,yama,bpf

$ lsmod | grep -E 'i915|xe'
xe                   4530176  3
i915                 5103616  0

All measurements dated 2026-08-16/17 were taken on this kernel (7.1.7, NixOS BUILD_ID 26.05.20260814.02e0898). The 2026-09-02 immediate-vs-regular command-list runs were taken after a routine system update — kernel 7.2.2, BUILD_ID 26.05.20260831.5dfba62 — on the same driver 26.18.38308.1 (the reproducer prints driver : 1.15.38308), with the same result.

Actual Behavior

In one paragraph. A kernel from a module the driver has switched to per-dispatch private memory gets its private surface from a reuse cache at every zeCommandListAppendLaunchKernel. Only the dispatch that allocates the surface adds it to the command list's residency container; every later dispatch just patches the surface's GPU address into cross-thread data. On an immediate command list the reuse cache belongs to the CSR (process lifetime) and the residency container is cleared after every flush, so after the first flush no submission ever lists the surface as resident again. Residency on VM_BIND platforms is lazy — a bound BO stays bound until evictUnusedAllocations() runs — and that sweep runs whenever a DRM_IOCTL_XE_VM_BIND fails. It unbinds every allocation that is not always-resident, not locked and not in flight; the private surface qualifies. Nothing re-binds it. From then on the kernel's private memory is at an unmapped GPU VA: on DG2 (VM created with scratch pages) reads return zeros, writes are dropped, and no error is reported anywhere, for the rest of the process. What an application sees is silent wrong results: correct output up to the first failed bind, wrong output forever after.

The single-file SYCL reproducer (linked under Source Code / Reproducer) does it in under ten seconds with a 1 GiB file, one device allocation and a warm page cache: 942 queue::memcpy calls totalling ~10 GiB, all reading the same region of a file-backed mmap — the copies whose userptr binds fail.

The defect, line by line (source at tag 26.18.38308.1; identical on master until 3d7a21dca9)

1. A module is put on the per-dispatch path when its kernels' private surfaces, summed, would not fit in device memory. ModuleImp::checkIfPrivateMemoryPerDispatchIsNeeded() (level_zero/core/source/module/module_imp.cpp:1404-1424) sums getPrivateSurfaceSize(perHwThreadPrivateMemorySize, computeUnitsUsedForScratch) over the module's kernels and sets allocatePrivateMemoryPerDispatch = modulePrivateMemorySize * numSubDevices > globalMemorySize. On an A770 computeUnitsUsedForScratch is 4096 (512 EUs × 8 threads, exact — see Additional Notes) and globalMemorySize is 16,225,245,593, so a module of 174 kernels at 22,784 B/thread (89 MiB each) crosses it. PyTorch's Shape.cpp module (~160 cat instantiations) is such a module. PrintDebugMessages=1 prints the decision: Private Memory Per Dispatch 1 for modulePrivateMemorySize … globalMemorySize ….

2. On that path the kernel object owns no private surface. KernelImp::initialize() (level_zero/core/source/kernel/kernel_imp.cpp:1313-1320) allocates a per-kernel surface only if false == module->shouldAllocatePrivateMemoryPerDispatch(), and pushes it into privateState.internalResidencyContainer — which appendLaunchKernelWithParams merges into the command list on every append (cmdlist_hw_xehp_and_later.inl:487-488, this->addResidency(internalResidencyContainer)). That is why modules under the threshold survive everything below: their surface is re-declared resident on every dispatch.

3. Per-dispatch surfaces come from a reuse cache instead. Every append calls allocateOrReuseKernelPrivateMemoryIfNeeded() (cmdlist_hw_xehp_and_later.inl:163). The immediate-command-list override (cmdlist_hw_immediate.inl:1821-1826) passes the CSR-owned cache, getCsr(false)->getOwnedPrivateAllocations() — one list per CSR, i.e. per process for a given queue, shared by every immediate command list on it, freed only in the CSR destructor. A regular command list passes its own ownedPrivateAllocations (cmdlist_hw.inl:4607-4612), freed on reset() and in its destructor (cmdlist_hw.inl:81-84, 141-144).

4. The failing line. CommandListCoreFamily::allocateOrReuseKernelPrivateMemory() (cmdlist_hw.inl:4615-4633):

   for (auto &alloc : privateAllocsToReuse) {
       if (sizePerHwThread == alloc.first) { privateAlloc = alloc.second; allocToReuseFound = true; break; }
   }
   if (!allocToReuseFound) {
       privateAlloc = kernelImp->allocatePrivateMemoryGraphicsAllocation();
       privateAllocsToReuse.push_back({sizePerHwThread, privateAlloc});
       this->commandContainer.addToResidencyContainer(privateAlloc);   // <-- only on the allocating dispatch
   }
   kernel->patchCrossthreadDataWithPrivateAllocation(privateAlloc);      // <-- every dispatch

   A reused surface is patched into the kernel's cross-thread data but never added to the residency container.

5. An immediate command list's residency container lives for one flush. executeCommandListImmediateWithFlushTaskImpl() (cmdlist_hw_immediate.inl:447-533) makes the container resident (makeResidentAndMigrate / makeResidentForResidencyContainer, lines 486/511) and then calls handlePostSubmissionState() (line 533), which is this->commandContainer.getResidencyContainer().clear() (cmdlist_hw.inl:244-246). So the surface is in exactly one residency set in the life of the process — the first flush's.

6. Binding is lazy, and so is unbinding. At submission CommandStreamReceiver::makeResident() stamps each listed allocation with the submission's task count (shared/source/command_stream/command_stream_receiver.cpp:226), and with DrmMemoryOperationsHandlerBind (what xe uses) makeResidentWithinOsContext() binds a BO only if bindInfo says it is not yet bound (drm_memory_operations_handler_bind.cpp:113-121). Nothing is unbound at submission time. Unbinding happens only in evictUnusedAllocations(), which is called from Drm::bindBufferObject() whenever a bind ioctl fails (shared/source/os_interface/linux/drm_neo.cpp:1792-1798):

   auto ret = changeBufferObjectBinding(this, osContext, vmHandleId, bo, true, forcePagingFence);
   if (ret != 0) {
       ...->evictUnusedAllocations(false, isAsyncFence);   // global sweep
       ret = changeBufferObjectBinding(...);               // retry
   }

   (Also from BufferObject::exec() on a failed execbuffer, drm_buffer_object.cpp:210,215, and from the ULLS-light retry in drm_command_stream.inl:143 — three routes to the same sweep.)

7. The sweep is global and the criterion is "not in flight". DrmMemoryOperationsHandler::evictUnusedAllocationsImpl() (shared/source/os_interface/linux/drm_memory_operations_handler.cpp:20-77) walks every sysmem and localmem allocation of the root device and evicts each one unless it isAlwaysResident, isLockedMemory, or isUsedByOsContext && getTaskCount(ctx) > *tagAddress. The private surface was stamped by the first flush, long completed, so it is a candidate; evictImpl() runs makeBOsResident(…, bind = false) — a VM_BIND unbind op — and marks it objectNotResident.

8. Nothing re-binds it. The next dispatch takes the same allocation from the CSR cache (step 4, allocToReuseFound == true), patches its VA into cross-thread data and submits. The allocation is not in the residency container, so makeResidentWithinOsContext() is never asked about it and its bindInfo == false is never acted on. The kernel runs with its private base at an unmapped VA. DG2 creates its VM with DRM_XE_VM_CREATE_FLAG_SCRATCH_PAGE (Drm::configureScratchPagePolicy() via isDisableScratchPagesSupported() == false before Xe2), so the access is absorbed: private reads return zeros, private writes are dropped, no page fault, no GPU error, every Level Zero call returns ZE_RESULT_SUCCESS.

What makes the binds fail in this report is incidental to the defect: on xe, a userptr VM_BIND of file-backed pages returns EPERM (512 times per run of the reproducer; anonymous or malloc sources never fail). But any failed bind — an -ENOMEM under memory pressure, which this machine's dmesg also shows on the xe_vm_bind_ioctl → xe_vma_userptr_pin_pages path — takes the same route to the same sweep, and the sweep is normal behaviour. The defect is purely that step 4 stops declaring a live surface resident.

What the regression commit intended, and what it missed. Before ee21f7c717 (2023-09-18), patchAndMoveToResidencyContainerPrivateSurface() pushed the surface into the kernel's residency container on every dispatch — which grows without bound, and the commit rightly stopped that and moved the declaration to the command container. It just gated it on allocation. The command container already deduplicates (removeDuplicatesFromResidencyContainer() at close() and at every immediate flush), so declaring on every dispatch costs nothing. git log -S 'addToResidencyContainer(privateAlloc)' on the file returns exactly that one commit; first release containing it: 23.39.27427.19; newest without: 23.35.27191.42.

Exactly which configurations are exposed

All four are required; each was varied independently (evidence below):

condition	how to tell	if absent
module on the per-dispatch path	NEOReadDebugKeys=1 PrintDebugMessages=1 … | grep 'Per Dispatch' prints 1	clean — per-kernel surface is re-declared every append (step 2)
a kernel in it with perHwThreadPrivateMemorySize > 0, dispatched again after the sweep	kernel properties private=…	a private=0 kernel in the same module is correct
immediate command list (CSR-owned reuse cache + per-flush residency)	UR/SYCL default on this platform; UR_L0_USE_IMMEDIATE_COMMANDLISTS=0 switches to regular	clean — see next table
a failed VM_BIND after the first flush completed	PrintBOBindingResult=1, strace shows DRM_IOCTL_XE_VM_BIND = -1 EPERM	clean — nothing ever unbinds the surface

Immediate vs regular command lists — new measurement, 2026-09-02, stock 26.18.38308.1, same binary (174 kernels), same file, PrintBOBindingResult=1 filtered to the 89 MiB private surface (size: 93323264), 3 runs each:

UR_L0_USE_IMMEDIATE_COMMANDLISTS	result	private-surface bind history
unset (default = immediate)	corrupt 3/3 (wrong 131072, untouched 65536)	bind BO-41 before run 1 → unbind BO-41 at the first failed bind → never re-bound; run 2 executes against the unmapped VA
1 (immediate)	corrupt 3/3	identical
0 (regular command lists)	clean 3/3	bind BO-36/unbind BO-36 around run 1; after the copies bind BO-41/unbind BO-41 at a new VA around run 2 — a fresh surface per submission, because a regular list's cache is freed on reset (step 3), so nothing is ever reused without being declared

Regular command lists pay an 89 MiB allocate-and-free per submission and are correct; immediate command lists reuse and are wrong. The same holds at the PyTorch level: xpu_cat_corruption_repro.py on the stock driver, same session, control (immediate) REPRODUCED cat=24576 on every rep; UR_L0_USE_IMMEDIATE_COMMANDLISTS=0 clean in 3/3 processes (9 reps, cat=0). That is the defect in one table, and it is why UR_L0_USE_IMMEDIATE_COMMANDLISTS=0 is a usable application-side workaround on unfixed drivers.

The fix

Declare the surface resident on every dispatch that uses it — which is what 3d7a21dca9 does on master, and what I built into 26.18.38308.1 and tested:

     if (!allocToReuseFound) {
         privateAlloc = kernelImp->allocatePrivateMemoryGraphicsAllocation();
         privateAllocsToReuse.push_back({sizePerHwThread, privateAlloc});
-        this->commandContainer.addToResidencyContainer(privateAlloc);
     }
+    this->commandContainer.addToResidencyContainer(privateAlloc);
     kernel->patchCrossthreadDataWithPrivateAllocation(privateAlloc);

Interleaved same-session A/B, each run straced to prove which libze_intel_gpu.so it opened:

arm	SYCL reproducer (174 kernels, file source)	PyTorch 2.13.0+xpu torch.cat reproducer
stock 26.18.38308.1	corrupt 3/3	corrupt 3/3 (cat=24576 wrong every rep)
same build + the two lines above	clean 3/3	clean 3/3

With the patch the bind trace gains the missing event: bind → unbind (at the failed bind) → bind again before the next dispatch — the same shape a module under the threshold already had. The SYCL column is deterministic (10/10 corrupt unpatched); the PyTorch column is stochastic and is corroboration.

In the real workload — one long-lived ComfyUI server on this A770, 30 image generations at a fixed seed, PrintBOBindingResult=1 counting events on the 89 MiB surface, each image scored by correlation against a known-good image at the same seed (correct 0.43-0.85, the failure 0.13-0.20, never overlapping):

driver	good images	private surface	shape of the session
stock	1 / 30	1 bind, 1 unbind	correct at generation 1, wrong from 2 to 30
patched	30 / 30	8 binds, 8 unbinds	every unbind followed by a re-bind; correct throughout

That is the symptom users see: it works, then at some point output turns to garbage, and only a restart helps.

Supporting evidence, condensed

Everything below was measured on this machine; the full detail is in the earlier revision of this issue if needed.

(a) Two debug keys move it in the predicted direction. MakeEachAllocationResident=2 (bind every allocation at flush): clean 3/3 where the default is corrupt 10/10. OverrideNumComputeUnitsForScratch=8192 pushes the otherwise-clean 128-kernel build over the threshold (Per Dispatch 1) and it becomes corrupt — kernel count and image size unchanged. =2048 puts 174 kernels under the threshold but undersizes the surface, so the kernel is wrong before any copies (wrong 65536, untouched 32768): 4096 is the real thread count.

(b) The per-dispatch flag is the discriminator, to the individual kernel. Adjacent kernel counts at three per-thread sizes, 2/2 each (globalMemorySize 16,225,245,593):

privateMemSize	kernels	modulePrivateMemorySize	Per Dispatch	after the copies
22,784 B	173 / 174	16,144,924,672 / 16,238,247,936	0 / 1	clean / corrupt
12,032 B	329 / 330	16,214,130,688 / 16,263,413,760	0 / 1	clean / corrupt
6,656 B	595 / 596	16,221,470,720 / 16,248,733,696	0 / 1	clean / corrupt

Not image size, not kernel count (595 clean at 6,656 B; 174 corrupt at 22,784 B). Per module: the same 174 kernels built with -fsycl-device-code-split=per_kernel give 0 per-dispatch modules and are clean 2/2. Free device memory is irrelevant (4 or 8 GiB of USM ballast changes nothing).

(c) Only file-backed copy sources fail their binds, and only failed binds matter. 174-kernel build, one variable at a time: file mmap (MAP_PRIVATE, MAP_SHARED, warm or cold cache, on disk or on tmpfs, one or 942 destination allocations) → corrupt; anonymous mmap or malloc source → clean, even at 120 GiB in 11,304 copies; allocations without copies → clean. strace -c: the file arms issue 512 × DRM_IOCTL_XE_VM_BIND = -1 EPERM and ~8,300 ioctls; the anonymous arm 0 EPERM and ~1,700 ioctls. The under-threshold file arm issues the identical 512 EPERM and is correct — the staging churn is harmless to a module whose surface is re-declared.

(d) Only privateMemSize > 0 kernels are wrong, and only in the per-dispatch module. The reproducer's in-module private=0 control (same module, same queue, same moment) scores 0 wrong while the private kernel scores 131072/131072. In a corrupted PyTorch process, of 20 ops checked against CPU only cat and stack are wrong (their kernel: private=22784, module prints Per Dispatch 1); index_select, gather, scatter, add, matmul, _foreach_add … all private=0, all correct.

(e) The KMD agrees, and nothing is migrated. xe tracepoints over the 2×2 (over/under threshold × file/anon), corrupting arm as in-run positive control: xe_bo_move vram→system 0, xe_vma_evict 0, xe_vma_invalidate 0 in every arm — the surface never leaves VRAM. xe_vma_unbind on the surface's range: corrupt arm 1 bind / 1 unbind; under-threshold file arm 2 / 2 (unbound by the same sweep, re-bound before its next dispatch); anon arms 1 / 1 at teardown. UMD PrintBOBindingResult and KMD tracepoints agree line for line. Sentinel guard allocations either side of the destination: 0 words disturbed while the same run reports REPRODUCED.

(f) Arguments are correct; it is silent; it is permanent and re-triggerable. The 1376-byte by-value argument decoded at zeKernelSetArgumentValue is byte-identical in corrupt and healthy processes; re-issuing it before the launch changes nothing. 118,650 launches / 39,735 copies / 31,855 syncs in one corrupted run, every one ZE_RESULT_SUCCESS; journalctl -k empty. Freeing everything, a fresh queue, 200 unrelated kernels: still broken. A module created after the copies is healthy (3/3) and is broken by a second round of copies (2/2) — any failed bind damages whatever qualifying modules are live at that moment. Two oneAPI toolchains (2026.0.0, 2026.1.0) and two PyTorch builds (2.13.0+xpu, 2.14.0.dev20260811+xpu) all reproduce on this driver.

Expected Behavior

A private allocation taken from the per-dispatch reuse cache must be added to the command list's residency container on every dispatch that uses it (as before ee21f7c717, and as 3d7a21dca9 now does), so that an evictUnusedAllocations() sweep can never leave a live kernel's private memory unmapped.

More generally: copying data to the device must not change the behaviour of already-loaded, unrelated modules, and a kernel must read back the private state it wrote. Silent zeros are the worst possible failure for a compute workload — plausible-looking, entirely wrong output with no diagnostic. If a private surface genuinely cannot be kept mapped, failing the launch would at least be visible.

Reproduction Rate

Always reproduces - 100%

Steps to Reproduce

# any file >= 512 MiB; its contents are never used
dd if=/dev/urandom of=/tmp/src.bin bs=1M count=1024

# make sure the Level Zero backend is what runs (a missing/bad driver silently
# falls back to OpenCL, which does not have the bug: the banner reads 26.18.0383xx
# instead of 1.15.383xx)
export ONEAPI_DEVICE_SELECTOR=level_zero:gpu

icpx -fsycl -O2 -DBLOAT_N=174 xpu_image_corruption_repro.cpp -o repro
./repro --file /tmp/src.bin --same-src --one-alloc --no-drop-cache            # exit 1: corrupt

# one kernel fewer -> module under the threshold -> per-kernel surface -> clean
icpx -fsycl -O2 -DBLOAT_N=173 xpu_image_corruption_repro.cpp -o repro173
./repro173 --file /tmp/src.bin --same-src --one-alloc --no-drop-cache         # exit 0

# same 174 kernels, regular command lists instead of immediate -> clean
UR_L0_USE_IMMEDIATE_COMMANDLISTS=0 \
./repro --file /tmp/src.bin --same-src --one-alloc --no-drop-cache            # exit 0

# same 174 kernels split one per module -> no per-dispatch module -> clean
icpx -fsycl -O2 -DBLOAT_N=174 -fsycl-device-code-split=per_kernel \
     xpu_image_corruption_repro.cpp -o repro_pk
./repro_pk --file /tmp/src.bin --same-src --one-alloc --no-drop-cache         # exit 0

# which side of the switch a build is on:
NEOReadDebugKeys=1 PrintDebugMessages=1 ./repro --file /tmp/src.bin \
    --same-src --one-alloc --no-drop-cache 2>&1 | grep 'Per Dispatch'

# the root cause from the driver's own logging: the 89 MiB private surface is
# bound once, unbound at the first failed VM_BIND, and never bound again
NEOReadDebugKeys=1 PrintBOBindingResult=1 ./repro --file /tmp/src.bin \
    --same-src --one-alloc --no-drop-cache 2>&1 | grep 93323264
#   bind BO-41 to VM 1,   ... size: 93323264, result: 0     <- before run 1
#   unbind BO-41 from VM 1, ... size: 93323264, result: 0   <- during the copies; nothing follows
# (repro173, or UR_L0_USE_IMMEDIATE_COMMANDLISTS=0: a bind precedes run 2)

# keeping every allocation resident avoids it entirely (clean 3/3)
NEOReadDebugKeys=1 MakeEachAllocationResident=2 ./repro --file /tmp/src.bin \
    --same-src --one-alloc --no-drop-cache                                    # exit 0

What the reproducer does:

1. Builds a module of BLOAT_N kernels that need private memory (privateMemSize=22784 each on this device; -DBATCH_SIZE varies it).
2. Runs one of them once, so the module exists and its surface is allocated and bound — and checks it: correct.
3. Issues 942 queue::memcpy calls, ~10 GiB, from an mmaped file into one device allocation. Their userptr binds fail with EPERM; the first failure runs the sweep that unbinds the surface.
4. Runs the same kernel again into an output buffer pre-filled with a sentinel, so "never written" and "written with zeros" are distinguishable (a fresh output buffer is recycled memory and misleads).
5. Prints the result and exits 1 if corrupted.

Expected output at BLOAT_N=174:

before upload : wrong 0, untouched 0   (private=0 control: wrong 0)
  10.04 GiB, 942 copies, 1 device allocations (4 s, 2481 MB/s)  [warm cache]
after upload  : wrong 131072, untouched 65536   (private=0 control: wrong 0)

The wrong/untouched counts are identical on every corrupting run. Observed rate: 10/10 consecutive runs in this form, 2/2 at each of the six boundary points, 3/3 in each arm of the immediate/regular and stock/patched A/Bs; once it fires it is permanent for the process.

Is this a regression?

• Yes, this is a regression - functionality that previously worked is now broken

Last Known Working Driver Version

23.35.27191.42 - newest tag not containing ee21f7c (git tag --contains); not run here: pre-xe-uAPI, will not zeInit on this machine

First Known Failing Driver Version

23.39.27427.19 - first tag containing ee21f7c (git tag --contains); reproduced here on 26.18.38308.1, which carries the same code; fixed on master by 3d7a21d (2026-08-27), not yet released

API Call Logs

Captured with a small dlsym interposer rather than unitrace (the UR Level Zero adapter dlopens libze_loader and resolves through dlsym, so plain LD_PRELOAD symbol interposition does not see the calls). Key results, all from corrupted processes:

[ze_count] launch=118650 err=0 | exec=... err=0 | copy=39735 err=0
           | evsync=31855 err=0 | fence=... err=0 | last_rc=0x0
[ze_count] setarg=... err=0 | indirect=... err=0 | allocdev=... err=0

The 1376-byte by-value argument decoded at zeKernelSetArgumentValue in a corrupted process, byte-identical to the healthy case:

[ze_argdump] arg1 sz=1376 zerobytes=715/1376 hash=3b65d4f7e8310d30
             output=ffffffffff870000
             input[0..1]=ffffffffff860000 ffffffffff861400
             offset[0..2]=0 5 12  dimSize[0..2]=5 7 11
             nElements[0..2]=1280 1792 2816
             -> launch groups 3 x 3 x 1

and the kernel properties that separate affected from unaffected kernels (this is PyTorch's cat, whose module is the one printing Per Dispatch 1; its five by-value arguments total 1424 B, of which the metadata struct above is 1376 B):

[ze_argdump] props args=5 local=0 private=22784 spill=0 maxSG=16  ...CatArrayBatchedCopy_alignedK_contig...
[ze_argdump] props args=8 local=0 private=0     spill=0 maxSG=32  ...VectorizedGatherKernel...
[ze_argdump] props args=3 local=0 private=0     spill=0 maxSG=32  ...VectorizedElementwiseKernel...AddFunctor...

Full logs available on request; I can re-capture with unitrace if you prefer that format.

strace Logs

The ioctl counts are under "Supporting evidence (c)" in Actual Behavior — they are a result, not a crash log. Full traces available on request. There is no failing syscall other than the 512 EPERM from DRM_IOCTL_XE_VM_BIND, which occur identically in a run that stays correct.

System Logs / dmesg Output

The corrupting runs are silent, and that is part of the report. Marking the journal, running three corrupting reproductions and reading back:

$ MARK=$(date '+%F %T'); ./repro ... ; ./repro ... ; ./repro ...   # 3x REPRODUCED
$ journalctl -k --since "$MARK"
-- No entries --

The machine's log over the week of this investigation is not empty, so here is what dmesg | grep -i -E 'i915|xe|drm|gpu' contains, with what each entry is. None of them is emitted by the reported flow:

[ 22521.463365] xe 0000:09:00.0: [drm] VM worker error: -12
[ 22531.553533] xe 0000:09:00.0: [drm] exec queue reset detected
...
[ 31669.172346]  handle_mm_fault+0xee/0x2f0
[ 31669.172346]  drm_gpusvm_get_pages+0x203/0x910 [drm_gpusvm_helper]
[ 31669.172359]  xe_vma_userptr_pin_pages+0xc2/0xd0 [xe]
[ 31669.172514]  vm_bind_ioctl_ops_parse+0x336/0x970 [xe]
[ 31669.172577]  xe_vm_bind_ioctl+0xd15/0x1bb0 [xe]
...
[112739.457009] xe ...: [drm] Tile0: GT0: Timedout job: seqno=4294967169, ... in benchdnn [295165]
[112739.511099] xe ...: [drm] Xe device coredump has been created
[211691.581057] python[545244]: segfault at 776f792a4000 ip 0000776eeb388469 \
                  error 4 in libze_intel_gpu.so.1.15.38308[788469,776eeac00000+ab2000]
[528778.547894] repro256[1538897]: segfault at 10 ip 000071448080c610 \
                  error 4 in libze_intel_gpu.so.1.15.38308[80c610,714480000000+ab2000]
[592707.436707] xe ...: [drm] Tile0: GT0: Engine memory CAT error: class=bcs, guc_id=6

• VM worker error: -12 / exec queue reset detected — deliberate VRAM-oversubscription tests (holding 15 GiB on a 16 GB card), not the reported flow.
• The xe_vm_bind_ioctl → xe_vma_userptr_pin_pages → drm_gpusvm_get_pages → handle_mm_fault stack — a host page-allocation failure during a userptr pin, from the same oversubscription work. Relevant only as a reminder that real -ENOMEM bind failures happen, and each one takes the same route into evictUnusedAllocations() as the EPERM in this report (step 6). Not tested as a trigger.
• Timedout job … in benchdnn — unrelated oneDNN microbenchmark.
• Two userspace segfaults in libze_intel_gpu.so.1.15.38308, symbolized: 0x80c610 is the first instruction of NEO::MultiGraphicsAllocation::getGraphicsAllocation(uint32_t) const with this == nullptr; 0x788469 is inside NEO::CommandStreamReceiver::baseWaitFunction reading the tag pointer at a mapped-looking address. Neither reproduces (four attempts on the first; the second happened once). Listed because they are in the attached log and inside your library; not claimed to be this bug — I will file separately if either can be reduced.
• Engine memory CAT error: class=bcs — one occurrence, copy engine, postdates every measurement here; unattributed.

Complete dmesg available on request.

Backtrace (if crash or hang occurred)

Not applicable — no crash and no hang in the reported flow. The application runs to completion and produces wrong numbers.

Source Code / Reproducer

One file, ~680 lines, SYCL only — xpu_image_corruption_repro.cpp, in a gist at https://gist.github.com/AIVirtuoso/0995fd491bd822f91cd6008e5fdf0ac3

No PyTorch, no model weights, no other dependency; the only external input is any file of at least 512 MiB, whose contents are never used.

It contains the kernel, the trigger and the check, and every control is a command-line flag, so each row of the tables above is one invocation:

--same-src     read the same region every copy (so a 1 GiB file suffices)
--one-alloc    one device allocation instead of 942
--no-copy      allocate but never copy
--host-src     copy from anonymous host memory instead of a mapping
--anon-mmap    copy from an anonymous mmap
--shared       MAP_SHARED instead of MAP_PRIVATE
--skip-upload  do nothing between the two kernel runs
--no-drop-cache  leave the page cache warm
--repeat N     N times the copy volume
--guard N      N MiB sentinel-filled guard allocations either side of the slab
--ballast G    hold G GiB of device USM for the whole run
--extra-lib    dlopen another module and check its kernel too
-DBLOAT_N=k    k private-memory kernels in the module (173 clean / 174 corrupt)
-DBATCH_SIZE=k sets privateMemSize (64→22784 B, 32→12032 B, 16→6656 B)

The kernel is a transcription of the concatenation kernel that first showed this in a real workload; the only thing that matters about it here is that IGC gives it privateMemSize=22784. The privateMemSize=0 in-module control is built in and reported on every run.

The same gist also contains xpu_cat_corruption_repro.py, the PyTorch-level form of the same fault (torch + safetensors only, generates its own 12.24 GiB file, ~2.5 min, stochastic). Happy to attach either file to this issue directly if you would rather have them here than in a gist.

Command Line / Application Details

./repro     --file /tmp/src.bin --same-src --one-alloc --no-drop-cache  # exit 1, corrupt
./repro173  --file /tmp/src.bin --same-src --one-alloc --no-drop-cache  # exit 0, correct
./repro_pk  --file /tmp/src.bin --same-src --one-alloc --no-drop-cache  # exit 0, correct

Runtime is under ten seconds each. Exit status is 0 when the kernel is still correct and 1 when the corruption reproduced, so it drops straight into a CI job.

oneAPI Version (if applicable)

Reproduces on both:

Intel(R) oneAPI DPC++/C++ Compiler 2026.0.0 (2026.0.0.20260331)
Intel(R) oneAPI DPC++/C++ Compiler 2026.1.0 (2026.1.0.20260617)

with matching intel-sycl-rt / intel-cmplr-lib-ur 2026.0.0 and 2026.1.0 respectively. Level Zero loader 1.28.5, IGC 2.34.4, gmmlib 22.10.0.

Screenshots / Video

No response

Additional Notes

On the checklist. Driver 26.18.38308.1 is the newest packaged in nixpkgs. The published 23.35 (intel-level-zero-gpu_1.3.27191.9) and 23.39 (1.3.27427.23) binaries were downloaded with matching libigdgmm12 and loaded via ZE_ENABLE_ALT_DRIVERS: they dlopen fine and then zeInit returns ZE_RESULT_ERROR_UNINITIALIZED — this A770 is on the xe KMD and those builds predate the settled xe uAPI (their linux/xe/ still issues DRM_IOCTL_XE_MMIO). So the regression boundary is established from history and from restoring the pre-commit semantics on current code (corrupt 3/3 → clean 3/3, plus the 30-generation session), not from running the old releases. An i915-bound card could bisect 23.35 vs 23.39 directly; I can boot this card on i915 if that is worth having.

Regression archaeology, for the reviewer. c06ddfc7b8 (2022-11-08) introduced the per-dispatch path, allocating and declaring residency every dispatch. 5807d512b3 (2023-08-31) introduced the reuse cache with the residency call still unconditional. 3b3e17e738 (2023-09-04) introduced the if (!allocToReuseFound) block but left residency outside it. ee21f7c717 (2023-09-18) moved it inside — the only commit git log -S 'addToResidencyContainer(privateAlloc)' returns on that file — and at ee21f7c717^ the kernel residency container was still copied into the command container on every append (cmdlist_hw_xehp_and_later.inl:357-363 at that commit), so reused surfaces reached every submission's residency set. 3d7a21dca9 (2026-08-27) moves it back out.

Application-side workarounds on unfixed drivers, each measured here:

workaround	effect	cost
UR_L0_USE_IMMEDIATE_COMMANDLISTS=0 (SYCL/UR)	regular command lists: no CSR-level reuse, surface allocated per submission — clean 3/3 on the SYCL reproducer and 3/3 processes on the PyTorch one (control reproduced in the same session)	an 89 MiB allocate/free per submission of an affected kernel; regular-list submission overhead
-fsycl-device-code-split=per_kernel	no module crosses the threshold, so no module is on the per-dispatch path — clean 2/2 at SYCL and PyTorch level (PyTorch's Shape.cpp: one 542,952 B image → 160 images, largest 6,588 B)	rebuild
NEOReadDebugKeys=1 MakeEachAllocationResident=2	every allocation bound at flush — clean 3/3	debug key, binds everything
avoid file-backed mmap as an H2D source	binds never fail, sweep never runs	not controllable from a framework; and any -ENOMEM bind failure reopens it

-fsycl-device-code-split is documented as not affecting correctness; that it does here is itself the argument that this is a driver defect.

Debug keys tried that change nothing (PrintDebugSettings=1 confirms each is read): EnableCopyWithStagingBuffers=0/1, TreatNonUsmForTransfersAsSharedSystem=0/1, EnableBOMmapCreate=0, EnableDeviceUsmAllocationPool=0, EnableUsmAllocationPoolManager=0. No key controls the per-dispatch switch itself (ForcePerDispatchPrivateMemorySize, AllocatePrivateMemoryPerDispatch and five similar names are unknown to the driver). MakeEachAllocationResident=1 hangs the reproducer before its first print — not investigated, mentioned in case it is unexpected.

Also ruled out by measurement: the by-value kernel argument (a hand-written kernel with a byte-identical 1424-byte argument set is correct in the same corrupted process, as long as its module is small); module size in bytes; kernel count; the allocations; the page cache; H2D volume; the device USM pool; memory pressure and occupancy; the hardware (30 min of saturating dense GEMM with retention checks is bit-exact, and a second process on the same GPU is correct while the first is corrupt).

Still open — three things I could not settle:

1. i915. The defect is above the KMD and the sweep is generic DRM code, so I expect it to be KMD-independent; but the trigger here is xe returning EPERM for a userptr bind of file-backed pages, and whether i915 refuses the same import, or whether DrmMemoryOperationsHandlerDefault (per-execbuf residency) is exposed at all, is untested.
2. Making it loud on DG2. DisableScratchPages=1 does not turn the unmapped access into a fault here: PrintXeLogs=1 confirms the flag reaches the ioctl (getFlagsForVmCreate 1,0,1, gemVmCreate f=0x2, no scratch-page bit) and the run still ends wrong 131072, untouched 65536 with an empty journalctl -k. Xe2+ default to no scratch pages, which may be why this only ever showed up on Arc A-series. If there is a way to make an unbound-VA access raise on DG2/xe, I will use it.
3. Two nits in the threshold arithmetic, neither a correctness issue once residency is fixed: checkIfPrivateMemoryPerDispatchIsNeeded() sums every kernel's whole-device surface as if all were resident at once (very pessimistic — 174 kernels of a 22 KB/thread kernel "exceed" a 16 GB card), and it reads the unaligned getGlobalMemorySize() (16,225,245,593) while checkIfThereIsSpaceForScratchOrPrivate() reads the page-aligned deviceInfo.globalMemSize (16,225,243,136).

How I ran into this. An image-generation workload (ComfyUI on this A770) started producing pure noise after loading a 12.5 GB checkpoint, with no error anywhere: a fresh process was almost always correct for its first generation and wrong for every one after — "it worked, then it started producing garbage, and only a restart helps". The silence was the expensive part; it took a long time to get from that symptom to a line number.

This issue was created with the help of an LLM.

[AIVirtuoso]

Update from the reporter, 2026-09-02.

The fix is on master. 3d7a21d "fix: add reused private memory to residency container" (2026-08-27, Related-To: GSD-13279) moves addToResidencyContainer(privateAlloc) out of the if (!allocToReuseFound) block in CommandListCoreFamily::allocateOrReuseKernelPrivateMemory() — byte for byte the change I built into 26.18.38308.1 and A/B-tested in this report (SYCL reproducer corrupt 3/3 → clean 3/3; PyTorch torch.cat reproducer corrupt 3/3 → clean 3/3; 30-generation application session 1/30 → 30/30 good). Its new unit test givenImmediateCmdListWhenPrivateAllocationIsReusedAfterResidencyContainerWasClearedThenAllocationIsMadeResidentAgain is exactly the failing case.

As of today it is not in any release (newest 26.31.39395.13, 2026-08-21) and not on releases/26.31 or releases/26.35. Could you

1. confirm that 3d7a21d is the fix for this issue,
2. backport it to the maintained release branches — it is a silent wrong-results regression present since 23.39.27427.19, and PyTorch on XPU hits it through its cat/stack kernels on every DG2 card,
3. and keep this open until a release carries it?

One new measurement that pins the exposure down further, now in the issue body. The bug needs an immediate command list: the reuse cache is the CSR's and the residency container is cleared after every flush (handlePostSubmissionState()), so after the first flush nothing ever declares the surface again. With regular command lists (UR_L0_USE_IMMEDIATE_COMMANDLISTS=0) the cache is per command list and freed on reset, a fresh surface is allocated per submission, and the same binary is clean 3/3 on the stock driver — PrintBOBindingResult=1 shows a bind before every dispatch — and the PyTorch torch.cat reproducer is clean in 3/3 processes against a control that reproduced in the same session. Immediate (the UR default, and =1 explicitly): corrupt 3/3, one bind, one unbind at the first failed VM_BIND, never re-bound. That is a usable application-side workaround on unfixed drivers, alongside -fsycl-device-code-split=per_kernel.

The issue body has been rewritten to lead with the line-by-line chain from that if to wrong results (eight steps, each with a file:line at 26.18.38308.1), the four conditions that must all hold, and the evidence condensed; the system information and reproducer are unchanged.

This comment was created with the help of an LLM.