[PW_SID:1067118] futex: Use runtime constants for futex_hash computation#1617
[PW_SID:1067118] futex: Use runtime constants for futex_hash computation#1617linux-riscv-bot wants to merge 8 commits intoworkflowfrom
Conversation
Futex hash computation requires a mask operation with read-only after init data that will be converted to a runtime constant in the subsequent commit. Introduce runtime_const_mask_32 to further optimize the mask operation in the futex hash computation hot path. [ prateek: Broke off the x86 chunk, commit message. ] Link: https://patch.msgid.link/20260227161841.GH606826@noisy.programming.kicks-ass.net Not-yet-signed-off-by: Peter Zijlstra <peterz@infradead.org> Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Futex hash computation requires a mask operation with read-only after init data that will be converted to a runtime constant in the subsequent commit. Introduce runtime_const_mask_32 to further optimize the mask operation in the futex hash computation hot path. GCC generates a: movz w1, #lo16, lsl #0 // w1 = bits [15:0] movk w1, #hi16, lsl #16 // w1 = full 32-bit value and w0, w0, w1 // w0 = w0 & w1 pattern to tackle arbitrary 32-bit masks and the same was also suggested by Claude which is implemented here. __runtime_fixup_ptr() already patches a "movz, + movk lsl #16" sequence which has been reused to patch the same sequence for __runtime_fixup_mask(). Assisted-by: Claude:claude-sonnet-4-5 Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
The current scheme to directly patch the kernel text for runtime
constants runs into the following issue with futex adapted to using
runtime constants on arm64:
Unable to handle kernel write to read-only memory at virtual address fff0000000378fc8
Mem abort info:
ESR = 0x000000009600004e
EC = 0x25: DABT (current EL), IL = 32 bits
SET = 0, FnV = 0
EA = 0, S1PTW = 0
FSC = 0x0e: level 2 permission fault
Data abort info:
ISV = 0, ISS = 0x0000004e, ISS2 = 0x00000000
CM = 0, WnR = 1, TnD = 0, TagAccess = 0
GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
swapper pgtable: 4k pages, 52-bit VAs, pgdp=00000000420a7000
[fff0000000378fc8] pgd=18000000bffff403, p4d=18000000bfffe403, pud=18000000bfffd403, pmd=0060000040200481
Internal error: Oops: 000000009600004e [#1] SMP
Modules linked in:
CPU: 1 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.19.0-rc6-00004-g7e6457d29e6a-dirty #291 PREEMPT
Hardware name: linux,dummy-virt (DT)
pstate: 81400009 (Nzcv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
pc : futex_init+0x13c/0x348
lr : futex_init+0xc8/0x348
sp : ffff80008002bd40
x29: ffff80008002bd40 x28: ffffa4b73ba0a160 x27: ffffa4b73bd10d74
x26: ffffa4b73cb68b28 x25: ffffa4b73ba0b000 x24: ffffa4b73c66b000
x23: 0000000000003fe0 x22: 0000000000000000 x21: ffffa4b73bd10d74
x20: 0000000000008000 x19: 0000000000000000 x18: 00000000ffffffff
x17: 000000007014db06 x16: ffffa4b73ca3ec08 x15: ffff80010002b937
x14: 0000000000000006 x13: fff0000077200000 x12: 00000000000002b2
x11: 00000000000000e6 x10: fff0000079e00000 x9 : fff0000077200000
x8 : fff00000034df9e0 x7 : 0000000000000200 x6 : ffffa4b73ba0b000
x5 : fff0000003510000 x4 : 0000000052803fe0 x3 : 0000000072a00000
x2 : fff0000000378fc8 x1 : ffffa4b739d78fd0 x0 : ffffa4b739d78fc8
Call trace:
futex_init+0x13c/0x348 (P)
do_one_initcall+0x6c/0x1b0
kernel_init_freeable+0x204/0x2e0
kernel_init+0x20/0x1d8
ret_from_fork+0x10/0x20
Code: 120b3c84 120b3c63 2a170084 2a130063 (29000c44)
---[ end trace 0000000000000000 ]---
The pc at "futex_init+0x13c/0x348" points to:
futex_init()
runtime_const_init(shift, __futex_shift)
__runtime_fixup_shift()
*p = cpu_to_le32(insn); /* <--- Here --- */
... which points to core_initcall() being too late to patch the kernel
text directly unlike the "d_hash_shift", "__names_cache" which are
initialized during start_kernel() before the protections are in place.
Use aarch64_insn_patch_text_nosync() to patch the runtime constants
instead of doing it directly to allow for running runtime_const_init()
slightly later into the boot.
Since aarch64_insn_patch_text_nosync() calls caches_clean_inval_pou()
internally, __runtime_fixup_caches() ends up being redundant.
runtime_const_init() are rare and the overheads of multiple calls to
caches_clean_inval_pou() instead of batching them together should be
negligible in practice.
At least one usage in kprobes.c suggests cpu_to_le32() conversion is not
necessary for aarch64_insn_patch_text_nosync() unlike in the current
scheme of patching *p directly.
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Futex hash computation requires a mask operation with read-only after init data that will be converted to a runtime constant in the subsequent commit. Introduce runtime_const_mask_32 to further optimize the mask operation in the futex hash computation hot path. GCC generates a: lui a0, 0x12346 # upper; +0x800 then >>12 for correct rounding addi a0, a0, 0x678 # lower 12 bits and a1, a1, a0 # a1 = a1 & a0 pattern to tackle arbitrary 32-bit masks and the same was also suggested by Claude which is implemented here. __runtime_fixup_ptr() already patches a "lui + addi" sequence which has been reused to patch the same sequence for __runtime_fixup_mask(). Assisted-by: Claude:claude-sonnet-4-5 Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Futex hash computation requires a mask operation with read-only after init data that will be converted to a runtime constant in the subsequent commit. Introduce runtime_const_mask_32 to further optimize the mask operation in the futex hash computation hot path. GCC generates a: nilf %r1,<imm32> to tackle arbitrary 32-bit masks and the same is implemented here. Immediate patching pattern for __runtime_fixup_mask() has been adopted from __runtime_fixup_ptr(). Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Add a dummy runtime_const_mask_32() for all the architectures that do not support runtime-const. Link: https://patch.msgid.link/20260227161841.GH606826@noisy.programming.kicks-ass.net Not-yet-signed-off-by: Peter Zijlstra <peterz@infradead.org> Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Runtime constify the read-only after init data __futex_shift(shift_32),
__futex_mask(mask_32), and __futex_queues(ptr) used in __futex_hash()
hot path to avoid referencing global variable.
This also allows __futex_queues to be allocated dynamically to
"nr_node_ids" slots instead of reserving config dependent MAX_NUMNODES
(1 << CONFIG_NODES_SHIFT) worth of slots upfront.
No functional chages intended.
[ prateek: Dynamically allocate __futex_queues, mark the global data
__ro_after_init since they are constified after futex_init(). ]
Link: https://patch.msgid.link/20260227161841.GH606826@noisy.programming.kicks-ass.net
Reported-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> # MAX_NUMNODES bloat
Not-yet-signed-off-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
|
Patch 1: "[RFC,v2,1/7] x86/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 1: "[RFC,v2,1/7] x86/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 1: "[RFC,v2,1/7] x86/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 1: "[RFC,v2,1/7] x86/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 1: "[RFC,v2,1/7] x86/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 1: "[RFC,v2,1/7] x86/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 1: "[RFC,v2,1/7] x86/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 1: "[RFC,v2,1/7] x86/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 1: "[RFC,v2,1/7] x86/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 1: "[RFC,v2,1/7] x86/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 1: "[RFC,v2,1/7] x86/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 1: "[RFC,v2,1/7] x86/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 2: "[RFC,v2,2/7] arm64/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 2: "[RFC,v2,2/7] arm64/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 2: "[RFC,v2,2/7] arm64/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 2: "[RFC,v2,2/7] arm64/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 2: "[RFC,v2,2/7] arm64/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 2: "[RFC,v2,2/7] arm64/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 2: "[RFC,v2,2/7] arm64/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 2: "[RFC,v2,2/7] arm64/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 2: "[RFC,v2,2/7] arm64/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 2: "[RFC,v2,2/7] arm64/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 5: "[RFC,v2,5/7] s390/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 6: "[RFC,v2,6/7] asm-generic/runtime-const: Add dummy runtime_const_mask_32()" |
|
Patch 6: "[RFC,v2,6/7] asm-generic/runtime-const: Add dummy runtime_const_mask_32()" |
|
Patch 6: "[RFC,v2,6/7] asm-generic/runtime-const: Add dummy runtime_const_mask_32()" |
|
Patch 6: "[RFC,v2,6/7] asm-generic/runtime-const: Add dummy runtime_const_mask_32()" |
|
Patch 6: "[RFC,v2,6/7] asm-generic/runtime-const: Add dummy runtime_const_mask_32()" |
|
Patch 6: "[RFC,v2,6/7] asm-generic/runtime-const: Add dummy runtime_const_mask_32()" |
|
Patch 6: "[RFC,v2,6/7] asm-generic/runtime-const: Add dummy runtime_const_mask_32()" |
|
Patch 6: "[RFC,v2,6/7] asm-generic/runtime-const: Add dummy runtime_const_mask_32()" |
|
Patch 6: "[RFC,v2,6/7] asm-generic/runtime-const: Add dummy runtime_const_mask_32()" |
|
Patch 6: "[RFC,v2,6/7] asm-generic/runtime-const: Add dummy runtime_const_mask_32()" |
|
Patch 6: "[RFC,v2,6/7] asm-generic/runtime-const: Add dummy runtime_const_mask_32()" |
|
Patch 6: "[RFC,v2,6/7] asm-generic/runtime-const: Add dummy runtime_const_mask_32()" |
|
Patch 7: "[RFC,v2,7/7] futex: Use runtime constants for __futex_hash() hot path" |
|
Patch 7: "[RFC,v2,7/7] futex: Use runtime constants for __futex_hash() hot path" |
|
Patch 7: "[RFC,v2,7/7] futex: Use runtime constants for __futex_hash() hot path" |
|
Patch 7: "[RFC,v2,7/7] futex: Use runtime constants for __futex_hash() hot path" |
|
Patch 7: "[RFC,v2,7/7] futex: Use runtime constants for __futex_hash() hot path" |
|
Patch 7: "[RFC,v2,7/7] futex: Use runtime constants for __futex_hash() hot path" |
|
Patch 7: "[RFC,v2,7/7] futex: Use runtime constants for __futex_hash() hot path" |
|
Patch 7: "[RFC,v2,7/7] futex: Use runtime constants for __futex_hash() hot path" |
|
Patch 7: "[RFC,v2,7/7] futex: Use runtime constants for __futex_hash() hot path" |
|
Patch 7: "[RFC,v2,7/7] futex: Use runtime constants for __futex_hash() hot path" |
|
Patch 7: "[RFC,v2,7/7] futex: Use runtime constants for __futex_hash() hot path" |
|
Patch 7: "[RFC,v2,7/7] futex: Use runtime constants for __futex_hash() hot path" |
dd35e73 to
da94c13
Compare
PR for series 1067118 applied to workflow
Name: futex: Use runtime constants for futex_hash computation
URL: https://patchwork.kernel.org/project/linux-riscv/list/?series=1067118
Version: 2