P2P Verification error

[vladie]

NVIDIA Open GPU Kernel Modules Version

595.71.05

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

ubuntu 24.04

Kernel Release

6.8.0-111

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

4x 3090

Describe the bug

my platform is gigabyte mz33-ar1 + amd epyc 9b45.
======== my grub kernel parameters:

GRUB_CMDLINE_LINUX_DEFAULT="ipv6.disable=1 mitigations=off amd_iommu=on iommu=pt"

======== I've installed driver without a problems, enabled Resizeable BAR in bios and i see this works fine:

lspci -vv | grep -A 5 "Physical Resizable BAR"
Capabilities: [bb0 v1] Physical Resizable BAR
BAR 0: current size: 16MB, supported: 16MB
BAR 1: current size: 32GB, supported: 64MB 128MB 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
BAR 3: current size: 32MB, supported: 32MB
Capabilities: [c1c v1] Physical Layer 16.0 GT/s Capabilities: [d00 v1] Lane Margining at the Receiver

Capabilities: [bb0 v1] Physical Resizable BAR
	BAR 0: current size: 16MB, supported: 16MB
	BAR 1: current size: 32GB, supported: 64MB 128MB 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
	BAR 3: current size: 32MB, supported: 32MB
Capabilities: [c1c v1] Physical Layer 16.0 GT/s <?>
Capabilities: [d00 v1] Lane Margining at the Receiver <?>

--
Capabilities: [bb0 v1] Physical Resizable BAR
BAR 0: current size: 16MB, supported: 16MB
BAR 1: current size: 32GB, supported: 64MB 128MB 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
BAR 3: current size: 32MB, supported: 32MB
Capabilities: [c1c v1] Physical Layer 16.0 GT/s Capabilities: [d00 v1] Lane Margining at the Receiver

Capabilities: [bb0 v1] Physical Resizable BAR
	BAR 0: current size: 16MB, supported: 16MB
	BAR 1: current size: 32GB, supported: 64MB 128MB 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
	BAR 3: current size: 32MB, supported: 32MB
Capabilities: [c1c v1] Physical Layer 16.0 GT/s <?>
Capabilities: [d00 v1] Lane Margining at the Receiver <?>

========== parameters to nvidia module i've tried:
options nvidia NVreg_RegistryDwords="ForceP2P=17;RMForceP2PType=1;RMPcieP2PType=1;PeerMappingOverride=1;RMForceStaticBar1=1"

and

options nvidia NVreg_RegistryDwords="ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1"

======= now i see in topology:

GPU0	GPU1	GPU2	GPU3	

GPU0 X OK OK OK
GPU1 OK X OK OK
GPU2 OK OK X OK
GPU3 OK OK OK X

============ now i trying to test P2P functionality with simpleP2P:

[./simpleP2P] - Starting...
Checking for multiple GPUs...
CUDA-capable device count: 4

Checking GPU(s) for support of peer to peer memory access...

Peer access from NVIDIA GeForce RTX 3090 (GPU0) -> NVIDIA GeForce RTX 3090 (GPU1) : Yes
Peer access from NVIDIA GeForce RTX 3090 (GPU0) -> NVIDIA GeForce RTX 3090 (GPU2) : Yes
Peer access from NVIDIA GeForce RTX 3090 (GPU0) -> NVIDIA GeForce RTX 3090 (GPU3) : Yes
Peer access from NVIDIA GeForce RTX 3090 (GPU1) -> NVIDIA GeForce RTX 3090 (GPU0) : Yes
Peer access from NVIDIA GeForce RTX 3090 (GPU1) -> NVIDIA GeForce RTX 3090 (GPU2) : Yes
Peer access from NVIDIA GeForce RTX 3090 (GPU1) -> NVIDIA GeForce RTX 3090 (GPU3) : Yes
Peer access from NVIDIA GeForce RTX 3090 (GPU2) -> NVIDIA GeForce RTX 3090 (GPU0) : Yes
Peer access from NVIDIA GeForce RTX 3090 (GPU2) -> NVIDIA GeForce RTX 3090 (GPU1) : Yes
Peer access from NVIDIA GeForce RTX 3090 (GPU2) -> NVIDIA GeForce RTX 3090 (GPU3) : Yes
Peer access from NVIDIA GeForce RTX 3090 (GPU3) -> NVIDIA GeForce RTX 3090 (GPU0) : Yes
Peer access from NVIDIA GeForce RTX 3090 (GPU3) -> NVIDIA GeForce RTX 3090 (GPU1) : Yes
Peer access from NVIDIA GeForce RTX 3090 (GPU3) -> NVIDIA GeForce RTX 3090 (GPU2) : Yes
Enabling peer access between GPU0 and GPU1...
Allocating buffers (64MB on GPU0, GPU1 and CPU Host)...
Creating event handles...
cudaMemcpyPeer / cudaMemcpy between GPU0 and GPU1: 25.19GB/s
Preparing host buffer and memcpy to GPU0...
Run kernel on GPU1, taking source data from GPU0 and writing to GPU1...
Run kernel on GPU0, taking source data from GPU1 and writing to GPU0...
Copy data back to host from GPU0 and verify results...
Verification error @ element 1: val = 0.000000, ref = 4.000000
Verification error @ element 2: val = 0.000000, ref = 8.000000
Verification error @ element 3: val = 0.000000, ref = 12.000000
Verification error @ element 4: val = 0.000000, ref = 16.000000
Verification error @ element 5: val = 0.000000, ref = 20.000000
Verification error @ element 6: val = 0.000000, ref = 24.000000
Verification error @ element 7: val = 0.000000, ref = 28.000000
Verification error @ element 8: val = 0.000000, ref = 32.000000
Verification error @ element 9: val = 0.000000, ref = 36.000000
Verification error @ element 10: val = 0.000000, ref = 40.000000
Verification error @ element 11: val = 0.000000, ref = 44.000000
Verification error @ element 12: val = 0.000000, ref = 48.000000
Disabling peer access...
Shutting down...
Test failed!

ik_llama, which tries to use P2P functionality - also outputs garbage.

============== with previous version of aikitoria open-gpu-kernel-modules in kernel log i see following:

[ 1107.918214] NVRM: nvAssertFailedNoLog: Assertion failed: ((base & RM_PAGE_MASK) == 0) @ kern_bus.c:298
[ 1107.918228] NVRM: nvAssertFailedNoLog: Assertion failed: remoteWMBoxLocalAddr != ~0ULL @ kern_bus_gm200.c:89
[ 1107.918234] NVRM: nvAssertFailedNoLog: Assertion failed: ((base & RM_PAGE_MASK) == 0) @ kern_bus.c:298
[ 1107.918239] NVRM: nvAssertFailedNoLog: Assertion failed: remoteWMBoxLocalAddr != ~0ULL @ kern_bus_gm200.c:89

but in latest driver version i've just installed this messages disappeared - but p2p not works.

how to fix this?

To Reproduce

use simpleP2P from cuda-samples.

Bug Incidence

Always

nvidia-bug-report.log.gz

none

More Info

No response

[vladie]

just tried old driver NVIDIA-SMI 570.211.01 from tinygrad repository - result are the same....

[vladie]

disabled IOMMU in BIOS and in GRUB kernel arguments but no luck - results isn't changed

[vladie]

also disabled SVM MODE in BIOS but no effect.

[vladie]

SR IOV == disabled also doesn't helped

[vladie]

SOLVED:
there are no need to disable IOMMU and use other options ever in kernel parameters.
the only two things need to do:
0) kernel parameters i use:
GRUB_CMDLINE_LINUX_DEFAULT="ipv6.disable=1 mitigations=off amd_iommu=on iommu=pt"

1. enable Resizable BAR support in BIOS (Advances -> PCI Subsystem Settings -> Re-Size BAR Support)
2. use following options for nvidia module in /etc/modprobe.d/nvidia/nvidia-graphics-drivers-kms.conf:

options nvidia NVreg_RegistryDwords="ForceP2P=17;RMForceP2PType=1;RMPcieP2PType=1;PeerMappingOverride=1;RMForceStaticBar1=1"

[vladie]

and i forgot to mention that i've reset BIOS to default settings - maybe that's why it started working

[R-omk]

Intel platform — P2P works after disabling intel_iommu

Tested on two Intel platforms with the same motherboard (ASUS PRIME Z890-P), both using open GPU kernel modules. P2P tests (simpleP2P, p2pBandwidthLatencyTest) pass successfully on both setups. Resizable BAR is enabled in BIOS.

Key finding: P2P only works with intel_iommu=off. Switching from intel_iommu=on → intel_iommu=off fixed the verification errors.

Setup 1: 4x RTX 3090 on PCIe Gen4 x16 per card
Setup 2: 4x RTX 5090 on separate switch, PCIe Gen5 x8 per card

GRUB (both systems):

GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=off iommu=pt mitigations=off"

nvidia.conf:

options nvidia NVreg_RegistryDwords="ForceP2P=17;RMForceP2PType=1;RMPcieP2PType=1;PeerMappingOverride=1;RMForceStaticBar1=1"

[frenzybiscuit]

You don't need any of this fancy stuff on AMD (7950x).

You just need to do the following for the 595.71.05 driver with Debian 13 kernel 6.17.2-1-pve:

amd_iommu=on iommu=pt

It works after this on that specific kernel.

[frenzybiscuit]

You do need to blacklist nouveau though.

[jtjstock]

Just for anyone who might wonder, this works on a 9700X on a consumer Asus Tuf Gaming B650-E WIFI board(Only has 1 PCIE slot on the CPU lanes, 2 M.2 slots on cpu, other pcie slots are chipset) with a frickin M.2 to PCI-E riser cable off of Ali express to connect the second GPU lol.. Set things up just as vladie did and it magically starts working

[vladie]

NOTE: if you have set up working P2P using instructions from my post above, and it stopped work after some time passed, clean your /lib/modules/ for all nvidia* modules and install driver again (also remove dkms nvidia* modules )