We have situation where when the userspace tries to bind RX/TX queues we see the following error in dmesg. The issue happens one in a while only when the user space application comes up. We have 64 RX/TX queues and run XDP program in a chain. Have we seen this kind of behaviour before?
[3138486.339874] ice 0000:51:00.0: Registered XDP mem model MEM_TYPE_XSK_BUFF_POOL on Rx ring 61
[3138486.364493] ice 0000:51:00.0: Registered XDP mem model MEM_TYPE_XSK_BUFF_POOL on Rx ring 62
[3138486.390071] ice 0000:51:00.0: Registered XDP mem model MEM_TYPE_XSK_BUFF_POOL on Rx ring 63
[3138492.635021] ------------[ cut here ]------------
[3138492.635024] NETDEV WATCHDOG: eth0 (ice): transmit queue 1 timed out
[3138492.635042] WARNING: CPU: 32 PID: 0 at net/sched/sch_generic.c:470 dev_watchdog+0x260/0x270
[3138492.635044] Modules linked in: bw_synproxy(OE) nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject overlay nf_conntrack_netlink bpf_preload iptable_filter bpfilter tls sch_fq udp_diag tcp_diag inet_diag ext2 dm_crypt encrypted_keys trusted dm_mod algif_skcipher af_alg loop nft_limit mousedev hid_generic usbhid cdc_ether usbnet mii nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 coretemp nft_counter nf_tables nfnetlink i10nm_edac skx_edac_common x86_pkg_temp_thermal kvm_intel kvm irqbypass ahci crc32_pclmul crc32c_intel libahci ghash_clmulni_intel xhci_pci i2c_i801 ice(OE) i2c_smbus aesni_intel intel_auxiliary(OE) xhci_hcd libata crypto_simd ptp cryptd evdev usbcore ioatdma glue_helper acpi_ipmi pps_core i2c_core scsi_mod usb_common dca ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter hwmon acpi_pad button binfmt_misc efivarfs [last unloaded: bw_synproxy]
[3138492.635099] CPU: 32 PID: 0 Comm: swapper/32 Tainted: G OE 5.10.251-249.983.amzn2int.x86_64 #1
[3138492.635100] Hardware name: PEGATRON PEGA1/ASRock Rack, BIOS L0.32 06/19/2025
[3138492.635102] RIP: 0010:dev_watchdog+0x260/0x270
[3138492.635104] Code: eb a9 48 8b 1c 24 c6 05 15 0f a7 01 01 48 89 df e8 e5 2d fa ff 44 89 e9 48 89 de 48 c7 c7 98 4b 3a 82 48 89 c2 e8 54 48 7e ff <0f> 0b eb 86 66 66 2e 0f 1f 84 00 00 00 00 00 90 0f 1f 44 00 00 41
[3138492.635106] RSP: 0018:ffffc90000c14eb8 EFLAGS: 00010286
[3138492.635107] RAX: 0000000000000000 RBX: ffff888806e0e000 RCX: ffff88fe3c21fbc8
[3138492.635108] RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff88fe3c21fbc0
[3138492.635109] RBP: ffff888806e0e3dc R08: 0000000000000000 R09: ffffc90000c14cf8
[3138492.635110] R10: ffffc90000c14cf0 R11: ffffffff83174068 R12: ffff890059cf8f40
[3138492.635110] R13: 0000000000000001 R14: ffff888806e0e480 R15: 0000000000000140
[3138492.635112] FS: 0000000000000000(0000) GS:ffff88fe3c200000(0000) knlGS:0000000000000000
[3138492.635113] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3138492.635114] CR2: 000004e13120f000 CR3: 0000000c9b7a4002 CR4: 0000000000770ee0
[3138492.635114] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3138492.635115] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[3138492.635116] PKRU: 55555554
[3138492.635117] Call Trace:
[3138492.635119] <IRQ>
[3138492.635124] ? shutdown_scheduler_queue+0x50/0x50
[3138492.635128] call_timer_fn+0x29/0x100
[3138492.635130] __run_timers.part.0+0x1d1/0x250
[3138492.635133] ? ktime_get+0x35/0xa0
[3138492.635134] ? clockevents_program_event+0x8a/0xf0
[3138492.635136] run_timer_softirq+0x26/0x50
[3138492.635139] __do_softirq+0xbc/0x26d
[3138492.635141] asm_call_irq_on_stack+0x12/0x20
[3138492.635142] </IRQ>
[3138492.635145] do_softirq_own_stack+0x37/0x50
[3138492.635149] irq_exit_rcu+0x8c/0xc0
[3138492.635153] sysvec_apic_timer_interrupt+0x36/0x80
[3138492.635155] asm_sysvec_apic_timer_interrupt+0x12/0x20
[3138492.635157] RIP: 0010:poll_idle+0x92/0xb1
[3138492.635159] Code: e8 5d 41 5c 41 5d 41 5e c3 cc cc cc cc 4c 89 f7 48 89 ee e8 10 d9 dd ff 49 89 c6 b8 c9 00 00 00 48 8b 13 83 e2 08 75 af f3 90 <83> e8 01 75 f1 65 8b 3d 2a e1 5e 7e e8 4d d1 6a ff 4c 29 e0 4c 39
[3138492.635160] RSP: 0018:ffffc90000523e68 EFLAGS: 00000246
[3138492.635161] RAX: 0000000000000078 RBX: ffff888101760000 RCX: 000000000000001f
[3138492.635162] RDX: 0000000000000000 RSI: 0000000037a6fccc RDI: ffd5ece717be7327
[3138492.635163] RBP: ffff88820080f400 R08: 000b267142a397d1 R09: 000000000000006a
[3138492.635164] R10: ffff88fe3c231844 R11: 0000000000000032 R12: 000b267142a2cc30
[3138492.635165] R13: 0000000000000000 R14: 0000000000989680 R15: 0000000000000000
[3138492.635169] ? poll_idle+0xa3/0xb1
[3138492.635172] cpuidle_enter_state+0x86/0x350
[3138492.635174] cpuidle_enter+0x29/0x40
[3138492.635177] cpuidle_idle_call+0x115/0x170
[3138492.635179] do_idle+0x7b/0xe0
[3138492.635181] cpu_startup_entry+0x19/0x20
[3138492.635184] secondary_startup_64_no_verify+0xb1/0xbb
[3138492.635186] ---[ end trace 07d249cc4aa05168 ]---
[3138492.635217] ice 0000:51:00.0 eth0: tx_timeout: VSI_num: 6, Q 1, NTC: 0x0, HW_HEAD: 0x19c5, NTU: 0x5, INT: 0x4000001
[3138492.635479] ice 0000:51:00.0 eth0: tx_timeout recovery level 1, txqueue 1
[3138493.051861] ice 0000:51:00.0: PTP reset successful
[3138500.419539] ice 0000:51:00.0: Registered XDP mem model MEM_TYPE_XSK_BUFF_POOL on Rx ring 0
[3138500.419547] ice 0000:51:00.0: Failed to allocate some buffers on UMEM enabled Rx ring 0 (pf_q 0)
[3138500.419693] ice 0000:51:00.0: Registered XDP mem model MEM_TYPE_XSK_BUFF_POOL on Rx ring 1
[3138500.419698] ice 0000:51:00.0: Failed to allocate some buffers on UMEM enabled Rx ring 1 (pf_q 1)
[3138500.419835] ice 0000:51:00.0: Registered XDP mem model MEM_TYPE_XSK_BUFF_POOL on Rx ring 2
[3138500.419840] ice 0000:51:00.0: Failed to allocate some buffers on UMEM enabled Rx ring 2 (pf_q 2)
[3138500.419983] ice 0000:51:00.0: Registered XDP mem model MEM_TYPE_XSK_BUFF_POOL on Rx ring 3
[3138500.419988] ice 0000:51:00.0: Failed to allocate some buffers on UMEM enabled Rx ring 3 (pf_q 3)
[3138500.420123] ice 0000:51:00.0: Registered XDP mem model MEM_TYPE_XSK_BUFF_POOL on Rx ring 4
[3138500.420127] ice 0000:51:00.0: Failed to allocate some buffers on UMEM enabled Rx ring 4 (pf_q 4)
[3138500.420269] ice 0000:51:00.0: Registered XDP mem model MEM_TYPE_XSK_BUFF_POOL on Rx ring 5
[3138500.420273] ice 0000:51:00.0: Failed to allocate some buffers on UMEM enabled Rx ring 5 (pf_q 5)
[3138500.420424] ice 0000:51:00.0: Registered XDP mem model MEM_TYPE_XSK_BUFF_POOL on Rx ring 6
We have situation where when the userspace tries to bind RX/TX queues we see the following error in dmesg. The issue happens one in a while only when the user space application comes up. We have 64 RX/TX queues and run XDP program in a chain. Have we seen this kind of behaviour before?